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FOREWORD 


Note  should  be  made  at  the  outset  that  the  original  title 
of  the  research  to  be  performed  under  Contract  No.  N00014-76-C-0163 
was  “Evaluation  of  Navy  Human  Relations  Training  for  Civilians.” 
Changes  in  the  form  and  content  of  human  relations  training  in  the 
Navy  have  occurred  since  the  inception  of  the  contract,  however,  so 
that  the  title  of  this  report  reflects  a broader  scope,  viz.,  human  rela- 
tions training  for  both  civilian  and  military  personnel  in  supervisory 
and  managerial  positions.  This  is  the  focus  of  the  Shore  Equal  Oppor- 
tunity Program  (SEOP),  the  program  discussed  in  the  case  study  section 
of  this  report. 


v 


TABLE  OF  CONTENTS 


FOREWORD  v 

PART  1:  CONCEPTUAL  MODEL  AND  PROCEDURES  FOR 

DEVELOPMENT  OF  AN  IMPACT  ASSESSMENT 3 

CHAPTER  I:  STAGES  IN  THE  DEVELOPMENT  OF  “HUMAN 
RELATIONS”  PROGRAMS  IN  THE  MILITARY 
DEPARTMENTS  5 

CHAPTER  II:  A MODEL  OF  THE  PROCESS  WHEREIN  INSTITUTIONAL 

DISCRIMINATION  ORIGINATES 9 

Organizational  Variables 11 

Individual  Variables  11 

Decisionmaking  Behavior  * 12 

Institutional-Level  Outcomes  ✓. 14 

Summary 15 

Implications  of  the  Model  for  Stage  Three  Training  Programs  ...  15 
Training  Program  Objectives  17 

CHAPTER  III:  PROGRAM  EVALUATION  19 

Policy  Statement  and  Identification  of  Objectives  21 

Identification  of  Appropriate  Impact  Criteria  and 

Criterion  Measures  22 

Selection  of  an  Evaluation  Design  26 

Assessment  and  Feedback  of  Impact  Data  30 

Measuring  Change  31 

Policy  Implications 33 

Research  Implications 33 

PART  11:  APPLICATION  OF  THE  MODEL-A  CASE  STUDY  35 

CHAPTER  IV:  HISTORY  OF  THE  SHORE  EQUAL  OPPORTUNITY 

PROGRAM 37 

vii 


CHAPTER  V:  APPROACH  TO  IMPACT  ASSESSMENT  45 

Assessment  of  Individual  Impacts  46 

Selection  of  a Design  46 

Selection  of  Appropriate  Impact  Measures  50 

Assessment  of  Institutional  Impacts  50 

CHAPTER  VI:  SUMMARY 61 


vui 


LIST  OF  FIGURES 


Figure  1.  Stages  in  the  Development  of  Human  Relations  Programs  6 

Figure  2.  Conceptual  Model  of  the  Process  wherein  Institutional 

Discrimination  Occurs  10 

Figure  3.  Stage  Three  Program  Objectives  17 

Figure  4.  Program  Impact  Evaluation  Model  20 

Figure  5.  Pre-Test/Post-Test  Control  Group  Design  26 

Figure  6.  Solomon  Four-Group  Design  27 

Figure  7.  Post-Test  Only,  Control  Group  Design  27 

Figure  8.  Comparison  of  Alternative  Program  Strategies  28 

Figure  9.  Non-Equivalent  Comparison  Group  Design 29 

Figure  10.  Time-Series  Experiment  29 

Figure  11.  Multiple  Time-Series  Design  29 

Figure  12.  Modification  in  Process  and  Content  Orientation  of  the 

“Human  Awareness  Course”  44 

Figure  13.  Examples  of  SEOP  Objectives 51 

Figure  14.  Illustrative  Example  of  a Type  I Display  for  a 

Hypothetical  Work  Unit  58 

Figure  15.  Illustrative  Example  of  a Type  II  Display  59 

Figure  1 6.  Illustrative  Example  of  a Type  III  Display  66 


ix 


EVALUATION  OF  NAVY  HUMAN  RELATIONS 
TRAINING  FOR  CIVILIAN  AND  MILITARY 
SUPERVISORS  AND  MANAGERS- 
A CONCEPTUAL  FRAMEWORK  AND  A CASE  STUDY 


Technical  Report 


PARTI 

CONCEPTUAL  MODEL  AND  PROCEDURES  FOR 
DEVELOPMENT  OF  AN  IMPACT  ASSESSMENT 


CHAPTER  I 


STAGES  IN  THE  DEVELOPMENT  OF  “HUMAN  RELATIONS”1 
PROGRAMS  IN  THE  MILITARY  DEPARTMENTS 


The  programs  now  in  existence  within  the  military  departments  which  deal  with 
human  relations  are  rather  more  sophisticated  efforts  to  deal  with  a complex  set  of  social 
issues  than  were  their  predecessors.  The  turbulent  social  conditions  of  the  late  1960’s  and 
early  1970  s prompted  the  military  to  recognize  and  respond  to  a real  need  to  improve  the 
state  of  relations  among  military  personnel  of  different  racial  and  ethnic  backgrounds.  The 
earliest  programs  established  for  this  purpose  were  assembled  on  a crash  basis,  with  little 
or  no  experience  in  the  area  to  guide  their  development.  They  grew  directly  out  of  wide- 
spread dissatisfaction  among  minority  group  members  over  the  treatment  they  were  receiving, 
both  from  whites  as  individuals  and  from  the  white-oriented  system.  This  dissatisfaction  led 
to  deep  hostility  toward  that  system  and  resulted  in  threatened  and  actual  interracial  violence, 
with  confrontations  occurring  both  between  minority  and  white  enlisted  personnel  and  be- 
tween minority  enlisted  personnel  and  the  military  authority  structure. 

The  earliest  of  the  military  race  relations  programs  focused  on  individual  racism, 
and  tended  to  operate  on  an  individualized  basis.  Minorities  aired  their  grievances,  from 
distant  past  to  the  present,  and  whites  were  compelled  to  listen,  perhaps  for  the  first  time. 
Interracial  confrontations  were  staged  or  created  in  a controlled  environment  with  the 
intent  of  increasing  white  awareness  about  the  status  and  treatment  of  minorities  in  our 
society  and  with  the  hope  of  increasing  the  feelings  of  guilt  among  whites.  This  would  lead, 
it  was  hoped,  to  changes  in  white  attitudes  toward  minorities,  with  consequent  improvement 
in  minority  status  and  treatment. 

This  early  method  for  dealing  with  race  relations  can  be  viewed  as  Stage  One  in  a 
developmental  process  which  continues  today  (see  Figure  1).  Stage  Two,  which  was  entered 
when  the  initial  crisis  period  had  receded  sufficiently  to  allow  a re-thinking  of  approach, 


1 As  used  in  this  document,  the  term  “human  relations”  is  restricted  to  the  topics  of  race  relations 
and  equal  opportunity. 


Figure  1 

Stages  in  the  Development  of  Human  Relations  Programs 


Stage  1— Race  Relations 

Stage  2— RR/EO 

Stage  3— EO 

Impetus 

• Minority  dissatisfaction  hos- 
tility, violence 

• Subjective  minority  dissatis- 
faction 

• Discriminatory  impacts  of  personnel 
decisions  on  minorities  and  women 

Mode  of 
Action 

• Confrontation  mode;  historical 
perspective  to  create  “white 
guilt” 

• Educational  mode 

• Training  mode 

Focus  of 
Program 

• Personal  racism,  individual 
experiences 

• Personal  racism  (and  sexism) 
plus  institutional  discrimi- 
nation 

• Institutional  race  and  sex  discrimi- 
nation 

Objectives 

• Attitude  change  hopefully 
leading  to  behavior  change 
in  the  long  run 

• Knowledge  increase,  attitude 
change,  ultimate  behavior 
change 

• Knowledge  increase,  perceptual 
change,  with  direct  implications 
for  behavior  change;  some  hope 
that  attitudes  will  change  in  the 
long  run 

Target 

Audience 

• General  military  population, 
especially  whites 

• General  population,  both 
military  and  civilian 

• Decisionmakers 

employed  slightly  different  objectives  and  methods.  This  second  stage  was  based  on  more 
of  an  “intellectual”  than  an  emotional  foundation.  A more  traditional  educational  mode 
of  operation  was  used  to  supplement  or  replace  confrontation  as  a learning  tactic.  The  scope 
of  the  programs  was  broadened  to  take  into  account  the  concept  of  institutional  discrimina- 
tion as  well  as  individual  discrimination.  The  objectives  of  Stage  Two  were  to  create  attitude 
change  and  increase  the  level  of  relevant  knowledge  in  the  general  population,  with  the  hope 
that  behavior  change  would  result  in  the  long  run.  Later  developments  of  the  Stage  Two 
model  have  included  sex  discrimination  as  well  as  racial,  and  have  become  increasingly  more 
concerned  with  eliciting  behavior  change,  with  or  without  attitude  change.  At  this  stage, 
also,  civilian  employees  of  the  military  began  to  be  included  in  the  target  audience. 

Stage  Three  in  the  process  is  now  beginning  to  develop.  This  stage  focuses  on 
institutional  discrimination,  and  the  impact  of  the  supervisor  or  manager  as  a decision- 
maker. The  underlying  assumption  is  that  every  personnel  decision  made  by  a supervisor 
or  manager  is  affected  by  equal  employment  opportunity  statutes  and  policies  and,  in  turn, 
has  implications  for  the  overall  status  of  EEO  within  the  organization.  (The  process  whereby 
this  occurs  is  described  in  Chapter  II.) 

Stage  Three  represents  a departure  from  previous  stages  on  several  additional  dimen- 
sions, also.  In  terms  of  objectives,  the  major  goal  of  a Stage  Three  program  is  to  change 
behavior— attitude  change  isnot  an  objective.  Another  clear  difference  is  that  Stage  Three 
programs  are  training  programs,  rather  than  traditional  educational  programs.  This  means 
that  in  Stage  Three,  the  sought-after  change  is  on  a job-related  skill  of  the  supervisor  or 
manager;  i.e.,  the  skill  of  making  personnel  decisions.  General  “awareness”  (of  historical 
events,  of  cultural  differences,  etc.)  in  the  Stage  Two  sense,  is  not  of  concern.  For  a Stage 
Three  program  to  be  successful,  it  must  create  the  conditions— knowledge,  perceptions,  and 
motivation— for  a supervisor  or  manager  to  make  every  personnel  decision  in  light  of  its 
equal  opportunity  implications.  Whether  the  motivation  is  to  do  what  is  “morally  right” 
or  simply  to  avoid  a lawsuit  is  not  of  direct  concern  to  the  program. 

The  categorization  of  human  relations  training  into  these  three  stages  may  be 
somewhat  artificial  in  that  no  specific  program  needs  to  be  a “pure”  example  of  any  type. 

Any  program  is  likely  to  emphasize  one  stage  or  another,  however. 
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The  Shore  Equal  Opportunity  Program  (SEOP),  the  focus  of  this  document,  has 
the  defining  characteristics  of  a Stage  Three  program  and  is,  in  fact,  one  of  the  very  few 
existing  examples  of  that  type. 
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CHAPTER  II 


A MODEL  OF  THE  PROCESS  WHEREIN 
INSTITUTIONAL  DISCRIMINATION  ORIGINATES 


Institutional  discrimination  is  the  central  concept  around  which  a Stage  Three 
program  is  organized.  It  is  impossible,  however,  to  modify  the  reality  of  institutional  dis- 
crimination without  dealing  directly  with  those  individuals  whose  behavior  creates  insti- 
tutional discrimination.  In  this  chapter,  a model  is  proposed  which  follows  the  process 
from  a single  decision  to  the  cumulative  picture  of  institutional  discrimination  in  the 
organization. 

Institutional  racial  discrimination  has  been  defined  as: 

...  a difference  in  what  happens  to  people  in  an  organization— 
a difference  which: 

( 1 ) is  correlated  with  skin  color; 

(2)  results  from  the  normal  functioning  of  the 
organization; 

(3)  operates  to  the  consistent  disadvantage  of 
persons  of  a particular  skin  color. ^ 

Institutional  discrimination  on  the  basis  of  race,  sex,  national  origin,  religion, 
age,  or  physical  handicap  is  the  central  focus  of  a Stage  Three  training  program.  The  par- 
ticipants in  a program  of  this  type  are  individual  supervisors  and  managers  who  have  the 
responsibility  for  making  a variety  of  personnel  decisions,  all  of  which  have  EEO  implications. 
Figure  2 depicts  a model  which  links  together  the  behavior  of  specific  individuals  with  insti- 
tutional outcomes. 

The  model  indicates  that  two  sets  of  conditions  comprise  the  environment  within 
which  decisions  are  made.  Organizational  variables  include  both  formal  and  informal  aspects 

2 

Peter  G.  NoidMe,  Measuring  Changes  in  Institutional  Racial  Discrimination  in  the  Army  (McLean, 
Va.:  Human  Sciences  Research,  Inc.,  March  1974),  p.  9. 
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Figure  2 


Conceptual  Model  of  the  Process  wherein 
Institutional  Discrimination  Occurs 


Conditions  Impacting  on 
the  Decisionmaker 


The  Decisionmaking  Process 
(Behavior) 


• Organizational  Variables  \ 

- Policy  \ 

- Procedures  \ 

• Long-Range  Planning 

- Available  Outcomes  \ 

• Short-Range  Planning 

- Norms  \ 

\ w 

• Utilization  and  Implementation 

of  Organizationally-Defined 

• Individual  Variables  / 

Procedures 

- Knowledge  / 

• The  Final  Decision 

- Perceptions 


Long-Range  Cumulative  Impacts  of 
Numerous  Decisions 


• Decision  Outcomes 


• Organizational  Race  and 
Sex  Statistics 


- Attitudes 


of  the  institution.  Individual  variables  are  those  conditions  inherent  in  the  individual  decision- 
making process.  The  cumulative  impacts  of  all  such  decisions  made  within  the  organization 
are  reflected  in  statistics  which  describe  the  makeup  of  the  organization  on  some  relevant 
dimension;  e.g.,  race,  sex,  age,  etc.  It  will  be  useful  to  look  at  each  element  of  the  model 
in  somewhat  greater  detail. 


Organizational  Variables 

Each  personnel  decision  is  made  within  some  organizationally-defined  framework 
consisting  of  formal  rules  and  regulations,  policy  guidelines  and  standing  operating  procedures 
(SOP’s);  in  most  cases,  the  range  of  available  outcomes  for  a given  type  of  decision  is  also 
defined  by  the  organization. 

In  addition  to  the  formal  aspects  of  the  decision  environment,  such  informal 
factors  as  peer  group  and  reference  group  norms  might  also  enter  into  a decision.  Each 
decisionmaker  has  been  socialized  into  the  organization,  has  learned  “do’s  and  don’t’s”  and 
shortcuts,  and  has  learned  which  behaviors  available  within  the  formal  structure  are  acceptable 
to  his  or  her  peers  and  organizational  superiors.  Thus,  the  full  range  of  options  might  never 
be  used  or  even  be  considered. 

Taken  together,  these  formal  and  informal  factors  delimit  the  organizational  en- 
vironment within  which  the  decisionmaker  operates. 

Individual  Variables 

A second  set  of  variables  determines  how  those  limits  are  perceived  and  observed. 
These  include: 

Knowledge— The  decisionmaker  can  only  comply  with  rules  and 
regulations  if  he  or  she  knows  what  they  are.  The  same  is  true  of 
SOP’s  and  of  informal  behavior  norms  as  well.  Equally  important 
is  that  the  supervisor  be  aware  of  the  objective  facts— as  opposed 
to  myths— concerning  the  treatment  and  status  of  minorities  and 
women  in  the  workforce  at  large  and  in  the  specific  organization. 
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Perceptions— The  individual’s  perception  of  reality  is,  for  that 
individual,  reality.  This  is  true  for  the  decisionmaker’s  perception 
of  the  formal  organizational  factors  described  above.  It  is  also 
true  of  that  individual’s  perception  of  what  constitutes  “appro- 
priate” roles  for  minorities  and  women,  and  of  the  “inherent” 
limitations  on  the  skills  and  abilities  of  these  groups. 

Attitudes— Even  in  a Stage  Three  program  where  attitude  change 
is  not  an  explicit  objective,  it  must  be  recognized  that  attitudes 
can  and  do  affect  decisions.  Such  attitudes  as  those  toward 
organizational  policy,  toward  EEO  objectives,  toward  working 
with  peers  who  are  of  minority  races,  toward  working  mothers, 
and  so  forth,  can  have  large  influences  on  decisionmaking  be- 
haviors and  on  decision  outcomes.  It  is  an  implicit  assumption 
in  a Stage  Three  program,  however,  that  knowledge  and  per- 
ceptual factors  can  outweigh  attitudes  in  determining  behavior. 


Decisionmaking  Behavior 

The  central  portion  of  the  model  deals  with  the  manager’s  behavior.  This  includes 
a variety  of  acts  which  lead  up  to  a decision  as  well  as  the  actual  “go-no  go”  decision.  It 
would  be  useful  at  this  point  to  consider  the  various  types  of  decisions  the  supervisor  or 
manager  participates  in  as  part  of  the  job.  Conversations  and  interviews  with  first-  and 
second-line  supervisors,  with  Navy  civilian  personnel  officers,  and  with  EEO  officers  have 
resulted  in  the  following  list  of  decisions  where  the  supervisor  or  manager  plays  a major  part, 
usually  the  primary  role,  and  sometimes  has  exclusive  authority. 

• Selection  of  personnel  to  fill  vacant  positions. 

• Decisions  concerning  advancement  of  an  individual  within  the 
organization. 

• Decisions  relating  to  job  reclassification,  either  upgrading  or 
downgrading. 

• Selection  of  individuals  for  specific  training  opportunities. 

• Selection  of  training  programs  for  specific  individuals. 
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• Utilization  of  the  skills  acquired  through  training. 

• Duty  assignments. 

• Disciplinary  actions. 

• Routine  evaluation  of  the  performance  of  subordinates. 

• Recommendations  for  awards,  rewards,  commendations, 
and  other  positive  acts. 

• Reduction-in-force  (RIF)  decisions. 

• Termination  of  employment. 

Once  again,  these  decisions  are  ordinarily  not  the  responsibility  of  the  supervisor 
alone.  Rules,  regulations,  and  SOP’s  apply.  The  personnel  office  is  involved  as  are  also  boards, 
panels,  and  committees  in  certain  of  the  decisions.  But  the  supervisor  can  have  the  major 
impact  in  every  case,  if  the  process  is  carried  out  as  specified. 

Take  long-range  planning,  for  example.  A manager  can  plan  a year  or  more  ahead 
by  looking  at  the  ages  of  his  or  her  employees  and  by  planning  in  advance  for  replacement 
of  retirees.  Also,  to  some  extent,  transfers  and  promotions  can  be  anticipated  in  advance. 

In  a more  immediate  sense,  the  short-range  planning  surrounding  a decision  can  be 
extremely  important.  If  the  supervisor  waits  until  a vacancy  occurs  and  then  decides  that  the 
vacancy  must  be  filled  immediately,  rather  than  either  planning  ahead  or  allowing  more  time, 
the  number  and  type  of  available  candidates  might  be  quite  limited.  It  has  been  said  that  the 
shorter  the  time  the  personnel  office  is  given  to  identify  candidates  for  most  vacant  positions, 
the  more  likely  the  candidates  are  to  be  white  males.  Given  greater  lead  time,  however,  quali- 
fied minorities  and  women  can  often  be  located. 

Similar  kinds  of  potential  poroblems  exist  in  regard  to  the  ways  in  which  other 
organizationally-defined  procedures  are  implemented,  such  as  keeping  interview  notes,  assign- 
ing “objective  scores”  to  candidates  for  qualifications,  etc. 
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The  final  decision,  then,  is  only  a small  segment  of  the  behavioral  sequence,  a 
segment  which  can  be  largely  predetermined  by  what  comes  before.  It  is,  however,  the 
element  that  gets  recorded  in  the  organization’s  statistics. 


Institutional-Level  Outcomes 


The  very  definition  of  institutional  discrimination  precludes  us  from  looking  at  it 
in  terms  of  individual  decisions.  Even  though,  in  fact,  personal  racism,  sexism,  etc.,  might 
have  entered  into  particular  decisions,  we  assume  that  any  overall  race  or  sex  imbalance  that 
exists  in  an  organization  can  best  be  approached  on  a systemic  basis.  Institutional  discrimina- 
tion is  only  evident  in  the  long-run,  cumulative  impacts  of  numerous  individual  decisions 
of  various  kinds  made  by  all  decisionmakers  in  the  organization  over  some  period  of  time. 

The  statistical  picture  of  the  composition  of  the  organization  is  made  up  of  the  outcomes 
of  these  individual  decisions. 

The  generic  types  of  statistics  referred  to  here  are  the  types  the  courts  are  looking 
to  in  class-action  lawsuits  which  charge  discrimination  under  Title  VII  of  the  Civil  Rights 
Act  of  1964  (as  amended  in  1972).  These  include: 

1 . Demographic  Statistics— these  compare  the  composition  of  the 
work  force  to  the  population  at  large  as  described  by  census  data' 
for  the  Standard  Metropolitan  Statistical  Area  (SMSA)  in  which 
the  employer  operates  for  locally-recruited  jobs,  and  by  national 
statistics  in  the  case  of  professional  and  highly-skilled  jobs  where 
the  national  market  serves  as  the  recruiting  area. 

2.  Concentration  Statistics— a measure  of  the  numerical  balance  of 
protected  class  members  in  comparison  to  other  employees  as 
they  are  distributed  throughout  the  organization;  e.g.,  concen- 
tration of  minorities  or  women  in  certain  job  assignments,  de- 
partments, or  work  areas. 

3.  Comparative  Statistics— a measure  of  the  rate  at  which  employees 
of  various  classes  are  able  to  take  advantage  of  the  benefits  of 

an  employment  situation.  E.g.,  if  50  percent  of  qualified  candi- 
dates for  promotion  are  black,  50  percent  of  promotions  should 
be  of  blacks,  even  if  only  two  percent  of  the  work  force  is  black. ^ 


^Source:  John  B.  Zimmerman  (Lt.,  USN),  “What  the  Courts  are  Saying,”  Advisor  (Fall  1976). 
Reprinted  in  Equal  Opportunity  Current  News,  113  (January  15, 1977),  pp.  7-11. 
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Summary 


In  summary,  the  model  describes  the  process  whereby  the  formal  and  informal 
environment  within  the  organization  interact  with  the  individual  decisionmaker’s  knowl- 
edge, perceptions,  and  attitudes  to  determine  the  way  that  individual  will  act  in  the  behavior 
sequence  leading  up  to  and  culminating  in  a decision.  The  impacts  of  these  individual  de- 
cisions are  aggregated  into  a statistical  picture  of  the  organization.  It  is  this  statistical 
picture,  showing  bias  in  a population  of  events,  upon  which  judgments  are  made  about  the 
presence  or  absence  of  institutional  discrimination. 


Implications  of  the  Model  for  Stage  Three 
Training  Programs 


What  kinds  of  impacts  can  be  expected  to  result  from  a training  program  of  the 
type  labeled  Stage  Three?  The  full  range  of  potential  impacts  is  somewhat  broader  than 
those  which  usually  are  stated  as  explicit  objectives.  In  speaking  of  potential  impacts,  the 
following  would  be  included: 

• Efforts  on  the  part  of  the  supervisor  or  manager  to  modify  the 
organizational  conditions  within  which  personnel  decisions  get 
made.  One  potential  effect  of  training  is  that  an  individual  might, 
as  a result  of  training,  attempt  to  change  some  aspect  of  the 
organization.  This  could  take  the  form  of  a lawsuit  to  change 
some  procedure  or  regulation,  or  it  might  be  an  informal  lobby- 
ing effort  to  change  a local  policy.  It  could  also  be  some  effort 
to  modify  the  norms  which  exist  concerning  EEO  among  the 
individual’s  peers  or  superiors.  It  might  even  be  in  the  form  of 
new  behavioral  guidelines  for  the  supervisor’s  employees  (although 
this  is  more  likely  to  be  a result  of  a Stage  Two  program  which 
places  more  emphasis  on  individual  discrimination).  These  poten- 
tial effects  could  conceivably  be  in  a negative  direction;  e.g.,  an 
effort  to  drop  the  policy  of  affirmative  action  because  the  indi- 
vidual perceives  it  as  “reverse  discrimination.”  However,  any 
such  effects  from  a successful  Stage  Three  program  would  be  in 
the  direction  favorable  to  EEO. 
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• Increases  in  knowledge  levels  on  topics  relevant  to  EEO.  Improved 
levels  of  factual  knowledge  and  understanding  of  the  nature  and 
objectives  of  EEO  should  be  expected  from  training.  The  super- 
visor should  go  away  from  the  program  knowing  more  about  what 
is  expected  and  required  of  supervisors,  what  the  organization’s 
policies  and  procedures  are  in  regard  to  EEO,  what  options  are 
available,  and  so  forth.  The  level  of  detail  will  depend  in  part  on 
the  objectives  of  the  particular  training  program,  but  might  include 
such  information  as  the  entry  requirements  for  a specific  training 
program  to  create  a vertical  career  path  for  an  employee,  or  the 
legal  requirements  for  keeping  concurrent  notes  on  the  process 

of  selecting  an  individual  to  fill  a vacancy. 

• Modification  of  the  trainee’s  perceptions  of  the  framework  within 
which  he  or  she  makes  decisions.  As  was  mentioned  earlier,  the 
individual’s  perception  of  reality  is,  for  that  individual,  reality. 

A Stage  Three  program  should  have  as  a goal  the  creation  of 
perceptions  which  match  objective  reality  as  closely  as  possible. 
This  might  include:  clearer  perception  of  the  individual  decision- 
maker’s impact  on  EEO;  more  realistic  perceptions  as  to  what 
constitute  artificial  ( versus  legitimate)  barriers;  a clearer  percep- 
tion of  the  difference  between  the  legally-available  options  and  the 
prevailing  norms;  i.e.,  what  is  possible  as  compared  to  “what  is 
done.”  Perceptual  change  on  numerous  dimensions  is  possible 
within  such  a program. 

• Change  in  the  attitudes  of  the  decisionmaker.  It  is  possible  that 
some  attitudes  will  change  as  a result  of  Stage  Three  training. 

We  need  not  specify  a mechanism  for  any  such  change.  It  might 
result  from  increased  knowledge  or  from  modified  perceptions, 
or  it  might  occur  simply  in  imitation  of  or  identification  with 

a particularly  charismatic  trainer.  We  need  only  assume  that  some 
attitudes  will  be  more  or  less  vulnerable  to  change,  and  that  change 
is  possible,  even  if  not  intended. 

• Behavioral  change  in  connection  with  the  decisionmaking  process. 
If  the  program  is  successful,  most  graduates  will  have  at  least  the 
requisite  knowledge  and  perceptions  to  allow  for  behavior  change 
in  the  desired  direction.  The  skills  will  be  available,  but  whether 
or  not  they  are  applied  appropriately  in  actual  practice  is  beyond 
the  scope  of  the  trainer  and  the  training  programs. 

• Increase  in  the  motivation  of  the  supervisor  or  manager  to  apply 
newly-acquired  principles  concerning  EEO.  A final  area  in  which 
program  impacts  might  possibly  be  felt  is  motivation  of  the 
graduate  to  put  into  practice  what  he  or  she  has  learned  in  the 
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training  course.  The  nature  of  that  motivation  may  vary  from 
person  to  person,  but  changes  in  motivational  level  might  well 
occur  within  the  structure  of  the  training  program. 


Training  Program  Objectives 

How  do  these  potential  impacts  of  a Stage  Three  training  program  relate  to  the 
explicit  and  implied  goals  of  such  a program?  By  definition,  the  ultimate,  long-range  goal 
of  a Stage  Three  program  is  to  effect  a change  in  the  statistical  picture  of  the  organization 
so  that  all  numeric  indicators  reflect  an  absence  of  institutional  discrimination.  The  more 
immediate  objective  is  to  modify  the  decisionmaking  behavior  of  each  individual  program 
participant  as  a means  to  achieving  the  ultimate  goal.  The  interim  objectives  seen  as  neces- 
sary to  the  desired  behavior  change  are  increased  knowledge  and  the  creation  of  realistic 
perceptions  to  enable  the  behavior  change  to  occur.  And  implicit  in  this  sequence  is  the 
goal  of  motivating  the  trainee  to  implement  his  upgraded  decisionmaking  skills  in  actual 
practice.  To  repeat,  attitude  change  is  not  an  explicit  goal,  nor  is  it  even  implied,  although 
it  would  be  welcomed  as  a serendipitous  side  effect.  It  can  also  be  assumed  that  no  effort 
will  be  directed  at  creating  change  in  the  formal  organizational  conditions,  and  that  no 
change  in  the  EEO-related  attitudinal  and  behavioral  norms  is  intended.  Figure  3 presents 
program  objectives  in  diagrammatic  form.  Keep  in  mind  that  the  achievement  of  institu- 
tional change  is  to  be  accomplished  through  the  medium  of  individual  behavior  change. 

Figure  3. 

Stage  Three  Program  Objectives 

Explicit  Implicit 

Instrumental  Objectives  Instrumental  Objective 

• Increased  Knowledge  Levels  • Motivation  to  Put  Upgraded 

decisionmaking  Skills  into 

• Realistic  Perceptions  Practice 

Interim  Objective 

* Change  in  Patterns  of  Behavior 
Employed  in  the  Decisionmaking 
Process 

Ultimate  Objective 

• Absence  of  Institutional 
Discrimination 
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CHAPTER  III 


PROGRAM  EVALUATION 


Now  that  the  types  of  programs  of  concern  here  have  been  described,  methods 
and  procedures  for  evaluating  the  impacts  of  such  programs  will  be  discussed. 

In  their  treatment  of  a proposed  policy  for  evaluation  of  federal  programs,  Wholey 
et  al. , identify  four  types  of  evaluation:  program  impact  evaluation;  program  strategy  evalu- 
ation; project  evaluation;  and  project  rating.4  The  first  of  these,  program  impact  evaluation, 
is  defined  as  . . assessment  of  the  overall  effectiveness  of  a . . . program  in  meeting  its 
objectives,  or  assessment  of  the  relative  effectiveness  of  two  or  more  programs  in  meeting 
common  objectives5’  (p.  25).  This  is  the  type  of  evaluation  we  will  be  concerned  with  here 
on  the  assumption  that  it  is  the  most  generally  applicable. 

These  same  writers  describe  evaluation  as  having  the  following  distinguishing 
characteristics. 


Evaluation  (1)  assesses  the  effectiveness  of  an  on-going  program 
in  achieving  its  objectives,  (2)  relies  on  the  principles  of  research 
design  to  distinguish  a program’s  effects  from  those  of  other 
forces  working  in  a situation,  and  (3)  aims  at  program  improve- 
ment through  a modification  of  current  operations,  (p.  23.) 

Figure  4 presents  a skeletal  model  of  the  evaluation  process.  Each  element  of  the 
model  is  discussed  below  in  conjunction  with  its  specific  application  to  a Stage  Three  EEO 
training  program. 

The  fundamental  impact  evaluation  questions  are: 

1.  Does  training  have  an  effect  on  those  individual  variables  which 
determine  decisionmaking  behavior? 

2.  Does  training  have  a direct  effect  on  decisionmaking  behavior? 

3.  Does  any  long-term  impact  on  indicators  of  institutional  dis- 
crimination occur  as  a result  of  training? 


4J.S.  Wholey,  J.W.  Scanlon,  H.G.  Duffy,  J.S.  Fukumoto,  and  L M.  Vogt,  Federal  Evaluation 

Policy : Analyzing  the  Effects  of  Public  Programs  (Washington,  D.C.:  The  Urban  Institute,  1973),  p.  24. 
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Figure  4. 

Program  Impact  Evaluation  Model 


1. 


Define  Program 
Policy 
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2.  3.  4.  5.  6. 


To  maximize  tne  likelihood  that  an  evaluation  of  program  impacts  will  result  in  answers  to 
these  questions,  several  conditions  must  be  met. 


Policy  Statement  and  Identification  of  Objectives 


First,  a clear-cut  statement  of  program  policy,  procedures,  and  objectives  must 
be  committed  to  writing.  Ideally,  this  will  be  accomplished  by  the  operating  agency,  but 
often  it  is  incumbent  upon  the  evaluator  to  produce  or  clarify  such  a statement,  especially 
in  regard  to  defining  and  clarifying  the  explicitly  recognized  program  objectives  and,  hope- 
fully, to  make  explicit  those  objectives  that  are  implied  by  the  selected  approach.  This  is 
done  in  the  first  two  steps  of  the  evaluation  model. 

For  a Stage  Three  program  we  have  identified  the  general  level  objectives.  How- 
ever, each  program  should  be  defined  in  terms  of  much  more  specific  and  detailed  objectives. 
For  example,  within  the  intermediate  objective  of  “modifying  decisionmaking  behavior” 
we  would  want  to  go  much  farther  toward  detailing  program  expectations,  to  include  such 
objectives  as: 

• inducing  the  supervisor  or  manager  to  perform  a review  of  his 
work  unit  to  identify  imminent  retirements,  training  needs, 
reclassification  needs,  etc.,  no  less  than  annually; 

• inducing  the  supervisor  or  manager  to  provide  the  personnel 
office  with  sufficient  lead  time  to  allow  identification  of 
qualified  female  and  minority  candidates  to  fill  a vacancy; 

• inducing  the  supervisor  or  manager  to  request  specifically,  in 
writing,  that  the  personnel  office  make  every  effort  to  provide 
women  and  minority  group  members  as  qualified  candidates  to 
fill  a vacancy;  etc. 


The  more  specific  and  detailed  the  objectives  are,  the  easier  it  is  to  accomplish 
the  succeeding  steps  in  the  evaluation. 
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Identification  of  Appropriate  Impact  Criteria 
and  Criterion  Measures 

Step  3 in  the  model  represents  criterion  development,  which  follows  closely  from 
identification  of  objectives.  In  the  case  of  the  ultimate  criterion  of  program  impact;  i.e., 
statistical  indicators  of  discrimination,  the  appropriate  measures  are  usually  obvious,  and 
include  measures  which  reflect  any  and  all  of  the  decisions  in  which  the  supervisor/manager 
take  part.  In  certain  intermediate  stages,  however,  the  specific  impact  criteria  to  be  measured 
may  be  less  obvious  if  objectives  are  not  detailed  enough.  Compare,  for  example,  the  relative 
problems  involved  in  measuring  achievement  of  the  same  objective  stated  in  two  different 
ways.  An  intermediate  objective  was  stated  above  as: 

• inducing  the  supervisor  or  manager  to  provide  the  personnel 
office  with  sufficient  lead  time  to  allow  identification  of 
qualified  female  and  minority  candidates  to  fill  a vacancy. 

There  are  two  obvious  drawbacks  to  this  statement  of  an  objective  in  regard  to  the  evaluation 
process.  One  is  use  of  the  term  “sufficient  lead  time”;  the  other  is  the  implication  that  quali- 
fied female  and  minority  candidates  will  be  available,  are  interested,  and  can  be  identified  if 
“sufficient”  time  is  allowed.  The  evaluation  question  is  one  which  can  be  answered  only  if  the 
operating  agency  provides  some  policy  decisions  upon  which  a more  specific  objective  state- 
ment can  be  based.  If  it  is  determined  that  one  month  is  an  “adequate”  amount  of  notice  in 
most  instances  to  allow  the  personnel  office  to  identify  female  and  minority  candidates  who 
do  exist  and  are  interested,  then  the  statement  of  this  objective  might  read: 

• inducing  the  supervisor  or  manager  to  allow  at  least  one  month’s 
time  for  the  personnel  office  to  identify  qualified  female  and 
minority  candidates  for  a vacant  position. 

The  implication  is  that,  if  no  such  candidates  are  identified  within  that  month,  the  supervisor 
need  delay  no  longer  and  may  proceed  to  select  from  among  the  candidates  available  at  that 
time,  even  if  all  are  white  males.  In  any  case,  it  is  assumed  that  every  effort  has  been  made 
to  adhere  to  sound  EEO  considerations. 
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If,  however,  the  operating  agency  will  be  satisfied  only  if  every  vacant  position 
has  at  least  one  female  and/or  one  minority  group  members  as  a viable  candidate,  it  may  be 
necessary  to  remove  any  time  limit.  The  objective  might  then  be  stated  as: 

• inducing  the  supervisor  or  manager  to  delay  filling  a vacant 
position  until  at  least  one  female  and/or  one  minority  group 
member  is  identified  as  a qualified  candidate. 

The  criterion  to  be  assessed  in  the  first  case  is  rather  straightforward;  i.e.,  is  there  a one-month 
period  between  the  opening  date  and  closing  date  of  the  search?  In  the  other  situation,  the 
criterion  is  equally  obvious,  but  quite  different:  Was  a qualified  minority  or  female  candidate 
considered?  The  two  are  equally  readily  measurable,  but  have  vastly  different  implications 
from  a policy  standpoint  and  from  the  perspective  of  the  supervisor,  who  has  a job  to  perform 
as  efficiently  and  effectively  as  possible. 

The  next  step  at  this  stage  in  the  evaluation  process  involves  the  translation  of 
criteria  into  specific  criterion  measures.  In  some  cases  appropriate  measures  will  already  be 
available  in  the  form  of  items,  scales,  and  instruments  developed  elsewhere,  but  this  should 
not  be  relied  on  as  a ready  source  of  criterion  measures.  The  number  of  such  measures  is 
not  large,  and  there  is  a high  likelihood  that  their  transferability  to  new  uses  will  be  quite 
limited,  especially  as  regards  measures  of  specific  cognitive  (knowledge),  perceptual,  and 
behavioral  objectives  of  Stage  Three  EEO  programs.  For  the  most  part  then,  new  measures 
will  have  to  be  created  for  measurement  of  these  individual  impact  criteria. 

Measures  of  the  impacts  of  training  on  the  individual  involve  a number  of  consid- 
erations. Certain  of  the  individual  impact  variables  (e.g.,  perceptions)  are  “internal  states” 
of  the  decisionmaker,  and  the  evaluator  must  rely  on  self-reports  as  the  major  data  source. 
These  self-reports  are  notoriously  susceptible  to  distortion,  either  deliberate  or  unconscious, 
as  a result  of  the  perceived  social  desirability  of  certain  responses.  The  term  “demand  char- 
acteristics” is  used  to  describe  the  dimensions  of  the  situation  which  the  respondent  per- 
ceives as  the  questioner’s  expectations  with  regard  to  the  respondent’s  answers.  A “test- 
wise”  respondent  might  select  the  “right”  answers  to  specific  questions  to  earn  a “good 
score”  on  a specific  scale  or  factor.  In  other  words,  he  or  she  will  say  what  the  questioner 
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“wants  to  hear,”  rather  than  answering  from  a purely  objective  standpoint.  This  can  and 
does  occur  without  the  respondent’s  awareness  that  it  is  happening;  or  it  might  be  done 
deliberately.  In  any  case,  the  evaluator  must  try  to  develop  attitude  and  perception  mea- 
sures which  are  subtle  and  which  minimize  demand  characteristics. 

A second  type  of  control  over  this  problem  is  to  seek  validation  of  the  respon- 
dent’s reported  “internal  states,”  either  from  outside  sources-observed  behaviors  or 
documentary  evidence,  for  example-or  by  means  of  a validity  scale  built  into  the  ques- 
tionnaire. 


Knowledge,  also  an  internal  state,  has  a separate  set  of  issues.  The  knowledge 
measure  must  be  reliable  (i.e.,  having  an  acceptably  small  error  component  which  is  not 
systematic  in  direction)  and  valid  in  terms  of  its  relevance  to  the  training  program’s  content 
and  objectives.  Specific  items  should  be  selected  so  as  not  to  be  so  easy  that  everyone  gets 
them  correct,  nor  so  difficult  that  nobody  does. 

Behavior  measures  can  be  either  relatively  easy  or  very  difficult  to  obtain.  Self- 
reports  of  behavior  or  behavior  intentions  are  subject  to  demand  characteristics.  Observation 
of  day-to-day  behavior  is  usually  quite  inefficient  and  uneconomical,  especially  for  infrequent 
events.  However,  there  are  other  appropriate  measures  for  some  behavior  impact  criteria 
relative  to  Stage  Three  programs.  These  may  be  of  the  type  described  earlier;  e.g.,  time  lags 
between  beginning  and  ending  of  a process  (filling  a vacant  position),  measurable  from 
records.  Or  they  may  be  obtainable  from  memoranda,  correspondence,  process  notes  main- 
tained during  a particular  decisionmaking  period,  etc.  The  evaluator’s  job  in  this  instance 
is  to  put  together  as  meaningful  a package  of  behavior  measures  as  possible  within  the  bounds 
of  the  available  budget  and  the  availability  of  accurate  and  readily  accessible  information. 

The  assessment  of  program  impacts  on  the  organization  as  a whole  is  based  on 
two  kinds  of  measures,  one  statistical,  the  other  more  of  a narrative  nature.  The  non- 
statistical  aspect  might  consist  of  periodic  status  reports,  management  reviews  of  organiza- 
tional practices,  “human  resources”  assessment  reports,  etc.  The  statistical  measures  must 
be  selected  to  reflect  the  outcomes  of  the  kinds  of  decisions  the  program  hopes  to  affect; 
i.e.,  decisions  about  hiring,  termination,  training,  assignments,  travel  status,  and  so  forth. 


24 


Earlier  (Chapter  II),  three  types  of  organizational  statistics  were  identified  which  are  used 
as  evidence  in  civil  court  suits  concerning  allegations  of  employment  discrimination.  The 
three  types  of  statistics  are  demographic,  concentration,  and  comparative  measures  of 
relative  presence,  absence,  or  maldistribution  of  protected  classes  of  individuals. 

The  interpretation  of  these  statistics  rests  in  the  concept  of  “expected  frequency”; 
i.e.,  what  the  statistics  would  look  like  if  arbitrary  discrimination  were  not  occurring?  This 
concept  requires  the  identification  of  an  appropriate  base  or  comparison  population  for 
each  separate  analysis. 

Take  as  an  example  the  relative  distribution  of  women  as  compared  to  men  in 
the  supervisory  levels  within  a specified  blue-collar  job  series.  On  a demographic  basis  the 
base  population  would  be  the  general  population  in  the  local  recruiting  area.  In  terms  of 
concentration  statistics,  that  job  series  might  best  be  viewed  in  comparison  with  the  percentage 
of  female  supervisors  in  certain  other  series;  e.g.,  clerical.  Finally,  comparative  statistics  might 
use  as  a base  population  the  number  of  women  who  are  eligible  for  promotion  to  supervisor 
in  that  series.  Each  type  of  statistic  shows  a somewhat  different  picture. 

Thus,  it  is  extremely  important  that  the  appropriate  comparison  be  made  for  each 
specific  purpose  and  that  an  appropriate  base  population  be  identified  for  determining  the 
expected  frequency  upon  which  a statistical  indicator  is  to  be  based. 

The  preceding  discussion  brings  out  the  need  for  the  evaluator  to  attend  to  mul- 
tiple data  sources  in  assessing  program  impacts.  A variety  of  sources  should  come  into 
consideration,  including,  but  not  necessarily  limited  to: 

• the  supervisor  or  manager; 

• that  individual’s  supervisor; 

• the  individual’s  subordinates; 

• statistical  data  banks; 

• records  and  files;  and 

• other  written  documentation. 

The  final  decision  as  to  which  of  these  sources  to  tap,  and  in  what  ways,  will  be  made  on 
the  basis  of  a variety  of  considerations,  both  scientific  and  practical. 
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Selection  of  an  Evaluation  Design 


The  selection  of  an  evaluation  design  represents  a critical  step  in  the  impact  assess- 
ment process  from  a scientific  point  of  view.  The  evaluation  design  is  the  plan  for  carrying 
out  the  collection  and  analysis  of  impact  data,  and  is  equal  in  importance  to  the  development 
of  relevant,  reliable,  and  valid  measures.  Both  are  essential  if  the  evaluation  is  to  provide 
maximally  usable  feedback;  the  value  of  results  is  directly  related  to  the  quality  of  the  impact 
measures  and  the  evaluation  design. 

The  ideal  situation  for  the  evaluator  is  one  in  which  he  or  she  can  develop  the  design 
prior  to  implementation  of  the  program  being  evaluated,  and  can  have  some  control  over  the 
relationship  between  the  program  and  the  impact  assessment.  The  best  type  of  design  for 
such  a situation,  in  the  opinion  of  Wholey,  et  al. ? is  that  referred  to  by  Campbell  and  Stanley*3 
as  a “true  experimental  design.”  Three  such  designs  are  available  for  evaluation  of  the  effects 
of  a single  program  strategy  and  one  for  comparing  alternative  strategies. 

The  first  of  the  single  strategy  evaluations  gives  a complete  assessment  of  program 
effects  alone,  and  is  referred  to  as  the  Pre-Test/Post-Test,  Control  Group  Design.  This  consists 
of  randomly  assigning  persons  to  either  an  experimental  group  which  experiences  the  training 
program  or  a control  group  which  does  not.  Criterion  measures  are  acquired  before  and  after 
training  for  the  experimental  group.  For  the  control  group  the  same  measures  are  taken  at 
the  same  times  as  for  the  experimental  group  except  that  no  treatment  (training)  occurs  in 
the  intervening  period.  Comparison  of  the  results  for  the  two  groups  allows  judgments  to  be 
made  about  the  impact  of  training.  Figure  5 is  a depiction  of  this  design. 

Figure  5. 

Pre-Test/Post-Test  Control  Group  Design 

Group*  Measurement  Treatment  Measurement 

Experimental  Pre-Test  Training  Post-Test 

Control  Pre-Test  Post-Test 

Assignment  to  groups  must  be  random  in  all  designs  discussed  here. 

^Wholey,  et  al.,  ibid.,  p.  88. 

^D.T.  Campbell  and  J.C.  Stanley,  Experimental  and  Quasi-Experimental  Designs  for  Research 
(Chicago:  Rand  McNally,  1963). 
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In  this  design,  not  only  can  the  effects  of  training  on  the  experimental  group  be 
detected  in  comparison  to  the  control  group,  but  also,  a “change  score”  (post-test  minus 
pre-test)  can  be  determined  for  the  experimental  group  and  compared  with  a similar  score 
for  the  control  group. ^ 

If  there  is  reason  to  believe  that  the  process  of  measurement  in  itself  will  affect 
the  results  obtained,  the  Solomon  Four-Group  Design  is  recommended.  If,  for  example,  it 
is  suspected  that  knowledge  scores  on  an  appropriate  measure  of  Stage  Three  program  effects 
will  be  influenced  by  the  pre-test  experience,  this  design  would  be  appropriate  because  it 
detects  such  effects.  It  can  be  represented  as  in  Figure  6. 

Figure  6. 

Solomon  Four-Group  Design 

Group  Measurement  Treatment 

Experimental  1 Pre-Test  Training 

Control  1 Pre-Test 

Experimental  2 — Training 

Control  2 — 


Measurement 

Post-Test 

Post-Test 

Post-Test 

Post-Test 


In  the  present  context  this  would  be  a valuable  design  if,  for  example,  it  were  feared  that 
respondents  would  actively  seek  out  the  answers  to  the  knowledge  questions  after  the 
initial  testing,  or  if  it  were  suspected  that,  having  answered  attitude  or  perception  questions 
once,  the  respondent  might  make  judgments  as  to  the  “right”  and  “wrong”  answers  (in  terms 
of  demand  characteristics),  and  systematically  modify  their  post- test  responses,  even  in  the 
absence  of  any  intervening  training. 

The  third  design  of  this  set  is  labelled  Post-Test  Only,  Control  Group  Design,  and 
is  represented  in  Figure  7. 


Figure  7. 

Post-Test  Only,  Control  Group  Design 

Group  Measurement  Treatment 

Experimental  - - - Training 

Control  — — 


Measurement 

Post-Test 

Post-Test 


n 

Change  score  analysis  is  a relatively  complex  matter,  and  the  analysis  of  raw  change  scores 
is  seldom  recommended.  See  discussion  later  in  this  chapter. 
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Note  that  here,  as  compared  with  the  pre-test/post-test  design,  change  score  analysis  is  not 
possible;  all  judgments  of  program  impact  being  based  on  comparison  of  post-test  scores. 

The  recommended  “true  experimental  design”  for  comparison  of  alternative 
program  strategies  is  represented  in  Figure  8. 


Figure  8. 

Comparison  of  Alternative  Program  Strategies 
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Treatment  Measurement 


Experimental  1 
Experimental  2 
Experimental  3 
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Training  Program  A 
Training  Program  B 
Training  Program  C 


Post-Test 
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This  is  an  extension  of  the  Pre-Test/Post-Test,  Control  Group  Design  for  a single  program  evalu- 
ation which  allows  for  comparison  of  the  effects  of  two,  three,  or  more  different  approaches 
to  accomplishing  program  objectives.  One  might  be  comparing  three  programs  based  on  a 
lecture  approach,  a small  group  guided  discussion  approach,  and  self-instruction,  respectively, 
for  example.  Once  the  most  efficacious  strategy  has  been  identified,  the  others  might  be 
abandoned.  Caution  is  advised,  however,  in  that  different  strategies  might  be  more  or  less 
effective  for  different  purposes;  e.g.,  lecture  for  knowledge  change  and  small  group  discussions 
for  perceptual  effects.  In  this  case,  a combination  program  might  be  the  best  bet. 

Remember,  all  of  these  designs  depend  upon  random  assignment  of  people  to  treat- 
ment groups,  and  this  is  not  always  an  available  option.  Where  random  assignment  is  not 
possible,  some  pre-  or  quasi-experimental  design  will  probably  prove  to  be  of  value.  Wholey, 
et  al .,8  refer  to  the  Non-Equivalent  Comparison  Group  Design  (Campbell  and  Stanley  Design  10, 
see  Figure  9),  as  “well  worth  using  in  many  instances  in  which  [the  three  true  experimental 
designs  recommended  for  single  program  evaluation]  are  impossible.”  Note  that  this  is  again 
the  basic  Pre-Test/Post-Test,  Control  Group  Design,  but  without  the  assurance  that  is  pro- 
vided by  random  assignment  that  the  groups  are  equivalent  before  the  program.  The  evaluator 
must  select  the  comparison  (control)  group  to  be  as  much  like  the  experimental  group  as  is 
possible  and  convenient,  on  relevant  dimensions. 

^Wholey,  et  al.,  op.  cit.,  p.  88. 
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Figure  9. 

Non-Equivalent  Comparison  Group  Design 


Group  Measurement  Treatment  Measurement 

Experimental  Pre-Test  Training  Post-Test 

Control  Pre-Test  — Post-Test 

In  many  cases  involving  human  relations  training,  it  will  be  necessary  to  employ 
this  design  or  one  of  two  time-series  designs.  The  Time-Series  Experiment  (Campbell  and 
Stanley  Design  7)  employs  several  pre-  and  post-test  measures  (see  Figure  10),  but  without 
control  group. 


Figure  10. 

Time-Series  Experiment 

Group  Measurement  Treatment  Measurement  Measurement 

Experimental  Pre-Test  Training  Post-Test  Followup 

A design  which  incorporates  the  best  features  of  these  last  two  is  the  Multiple 
Time-Series  Design  (Campbell  and  Stanley  Design  14),  as  depicted  in  Figure  1 1. 

Figure  1 1 . 
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Finally,  where  the  program  is  in  operation  before  the  evaluation  design  is  selected, 
the  possibility  of  taking  a pre-test  measure  is  precluded  and  no  appropriate  control  group  is 
available,  the  One-Shot  Case  Study  Design,  despite  its  decided  shortcomings,  may  be  all  that 
is  available. 
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The  need  for  “true  experimental  designs,”  control  groups,  and  pre-  and  post-test 
comparisons  might  not  be  obvious  to  one  whose  interest  is  only  in  starting  a program  and 
keeping  it  running.  This  would  be  the  person  who  is  interested  in  process  evaluation,  and 
who  wants  to  know,  more  than  anything  else,  how  to  run  the  program  most  efficiently. 

But  program  evaluation  has  the  capability  to  provide  much  more  information  than  that,  when 
used  judiciously. 

The  policymaker  can  use  the  evaluation  results  as  a basis  for  making  decisions  about 
the  future  course  of  the  program.  Those  in  charge  of  day-to-day  operation  of  the  program 
can  learn  what  works  and  what  doesn’t  from  immediate  feedback  of  evaluation  results.  And 
the  individual  with  a research  orientation  can  explore  certain  hypotheses  about  training 
effects  in  an  effort  to  improve  program  operations  and,  as  a result,  program  effectiveness. 

In  short,  one  stands  to  gain  a lot  by  a scientific  approach  to  program  evaluation. 

It  is  essential  that  the  soundest  evaluation  design  available  under  the  circumstances  be  em- 
ployed, and  that  planning  for  evaluation  begin  during  the  early  program  planning  stages. 

Assessment  and  Feedback  of  Impact  Data 

Once  the  program  has  been  implemented  and  impact  data  acquired  in  accordance 
with  the  design  selected,  what  does  one  look  for  in  the  data?  Obviously,  the  program  can 
be  considered  successful  if  changes  in  the  desired  direction  occur  with  regard  to:  behavior; 
knowledge;  perceptions;  and  indicators  of  EEO  progress  at  the  institutional  level.  The 
questions  to  be  answered  include: 

1.  Has  change  occurred? 

2.  Is  the  change  sufficient  to  justify  the  expenditures  involved  in 
the  program? 

3.  Is  the  change  a persistent  one;  i.e.,  does  it  hold  up  over  time, 
or  is  it  fleeting  and  momentary? 

The  first  of  these  is  an  evaluation  question;  the  second  a policy  question;  and  the  third  a 
research  question. 
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Measuring  Change 


Answering  the  basic  question  as  to  whether  or  not  any  change  in  the  dependent 
measures  occurs  as  a result  of  training  can  be  a very  complex  process.  The  professional 
literature  on  measurement  of  change  scores  is  substantial,  and  the  evaluator  must  take  a 
number  of  technical  considerations  into  account.  A recent  paper  by  Hummel-Rossi  and 
Weinberg^  summarizes  the  major  issues  in  change  score  analysis  and  provides  guidelines 
for  the  measurement  of  change  in  various  situations  and  for  analysis  of  the  statistical  sig- 
nificance of  the  measured  change. 

The  major  problems  these  writers  identify  in  the  use  of  raw  gain  scores;  i.e.,  post- 
test score  minus  pre-test  score,  are  as  follows: 

1 . low  pre-test  and  post-test  reliabilities; 

2.  the  regression  to  the  mean  phenomenon; 

3.  the  need  to  examine  the  relation  between  pre-test  score  and 
change  score ; 

4.  pretest-and  post-test  measures  in  different  metrics. 

Of  these  four,  those  most  likely  to  occur  in  the  context  under  discussion  here  are  the  first 
two,  low  reliability  and  regression  to  the  mean. 

The  evaluator  can  reduce  the  problems  created  by  low  pre-  and  post-test  reliabilities 
by  creating  measurement  instruments  which  have  high  reliabilities.  This  does,  certainly,  re- 
quire the  foresight  to  allow  adequate  time  for  instrument  development,  with  several  pre-tests 
to  eliminate  sources  of  unreliability;  or  alternatively,  the  selection  of  pre-existing  measures  of 
acceptable  levels  of  reliability.  Regression  to  the  mean,  however,  can  occur  with  any  measure 
where  respondents  can  earn  extremely  high  or  extremely  low  scores;  i.e.,  virtually  any  of  the 
measures  discussed  here. 

Hummel-Rossi  and  Weinberg  discuss  alternatives  to  the  use  of  raw  gain  scores, 
including:  estimate  of  true  gain  score;  a base-free  measure  of  change  obtained  by  removing 

Q 

Barbara  Hummel-Rossi  and  Sharon  L.  Weinberg,  “Practical  Guidelines  in  Applying  Current 
Theories  to  the  Measurement  of  Change.  Part  I:  Problems  in  Measuring  Change  and  Recommended  Pro- 
cedures,” Catalog  of  Selected  Documents  in  Psychology , 5 (1975),  p.  226. 
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that  portion  of  the  post-test  score  linearly  predictable  from  the  pre-test  score  which  removes 
the  true  pre-test  score  from  the  true  post-test  score;  and  use  of  partial  correlation  or  multiple 
regression  techniques.  The  authors  recommend  the  last  two  approaches  for  most  cases. 

They  give  examples  of  the  application  of  analytic  methods  to  several  situations,  including: 

1 . a one-group  design,  without  control; 

2.  two  or  more  groups,  with  random  assignment  of  individuals; 
and 

3.  two  or  more  groups,  with  members  not  randomly  assigned. 

Other  designs  are  discussed,  also,  but  they  are  less  likely  to  occur  within  the  context  being 
discussed.  The  analytic  procedures  recommended  for  these  designs  are,  respectively:  raw 
gain  score  analysis;  analysis  of  covariance ; and  large  sample  covariance  analysis. 

These  approaches  are  appropriate  for  most  of  the  designs  acceptable  for  program 
evaluation  purposes  when  applied  to  measures  of  individual  change.  One  method  of  analyzing 
institutional  change  is  by  use  of  an  index  called  the  Difference  Indicator  (D.I.),  developed 
under  an  Army  Research  Institute  (ARI)  contract  with  Human  Sciences  Research,  and  de- 
scribedin  several  research  reports,  the  most  recent  of  which**-*  describes  a method  for  assess- 
ing the  significance  of  differences  in  the  D.I. 

The  D.I.  is  based  on  the  concept  of  expected  frequency,  as  described  earlier;  i.e., 
what  one  would  expect  a particular  index  of  race  or  sex  representativeness  to  look  like  if, 
in  fact,  discrimination  were  not  occurring.  A more  detailed  description  of  the  D.I.  appears 
in  Chapter  V of  this  report. 


***Peter  G.  Nordlie  and  William  S.  Edmonds,  Commanders’  Handbook  for  Assessing  Institutional 
Racial  Discrimination  in  Units  (McLean,  Va.:  Human  Sciences  Research,  Inc.,  February  1977).  (In  Review.) 
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Policy  Implications 


Decisions  concerning  the  continuation  of  a program,  from  a policy  standpoint, 
will  be  based  on  an  assessment  of  what  results  were  obtained  for  the  investment  made. 

Impact  assessment  results  are  designed  to  provide  information  on  which  to  base  an  answer 
to  this  question.  It  is  up  to  the  evaluator  to  see  that  it  is  sufficient  for  that  purpose  and 
that  the  policymakers  have  the  information.  No  one  can  guarantee  that  policymakers  will 
use  the  results  to  best  advantage,  and  that  is  often  beyond  the  scope  of  the  evaluator’s 
responsibilities. 

The  decisions  involved  will  revolve  around  the  issues  of  the  practical  consequences 
of  changes  that  are  observed  and  the  cost  to  obtain  those  consequences.  In  some  instances, 
even  apparently  minute  changes  in  the  desired  direction  may  have  practical  significance  and 
may  justify  relatively  large  expenditures.  In  other  situations,  policymakers  might  demand 
evidence  of  massive  change  for  even  relatively  minor  investment  before  they  will  be  convinced 
of  the  good  to  be  derived  from  the  program.  About  the  best  to  be  hoped  for  is  that  the  evalu- 
ation results  will  be  accurately  reflective  of  reality  and  that  they  will  be  easy  to  understand 
and  interpret.  Ease  of  interpretation  is  a decided  advantage  of  the  Difference  Indicator 
mentioned  above,  when  displayed  graphically.  The  same  cannot  always  be  said  of  tests  of 
statistical  significance,  unfortunately. 

Research  Implications 

Finally,  the  duration  of  change  is  a research  question.  The  phenomenon  of  the 
“sleeper  effect,”  in  which  immediate  change  is  small  but  change  continues  to  build  over 
time,  and  the  “decay”  phenomenon,  in  which  immediate  change  occurs,  but  is  soon  followed 
by  a return  to  pre-training  levels  of  performance,  are  but  two  aspects  of  RR/EO  training 
which  have  not  been  thoroughly  studied.  One  would  certainly  want  to  know,  for  example, 
the  optimum  time  between  the  initial  training  experience  and  some  “refresher  course”  or 
other  followup  technique.  The  research  aspects  of  evaluation  design  will  allow  for  such 
issues  as  these  to  be  studied,  with  the  goal  of  improving  future  program  results. 
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PART  II 

APPLICATION  OF  THE  MODEL- 
A CASE  STUDY 


CHAPTER  IV 

HISTORY  OF  THE  SHORE  EQUAL  OPPORTUNITY  PROGRAM 


The  objectives  of  the  contract  under  which  the  work  described  here  was  performed 
were  originally  stated  as  follows: 

1.  Determine  the  total  set  of  training  objectives  for  Navy  human 
relations  training  programs  for  civilian  personnel. 

2.  Develop  criterion  measures  which  reliably  reflect  the  extent 
each  of  the  training  objectives  is  achieved. 

3.  Assess  the  impact  of  Navy  human  relations  training  programs 
for  supervisors  of  civilians  utilizing  the  criterion  measures 
developed,  and  an  appropriate  research  design. 

4.  Explore  the  relationships  between  various  independent 
variables  and  the  criterion  measures. 

5.  Recommend  changes  in  the  training  programs  based  on  the 
assessment  results. 

At  the  outset,  the  focus  of  these  efforts  was  understood  to  be  a training  program 
then  under  development  by  the  Navy’s  Office  of  Civilian  Manpower  Management  (OCMM)  * * 
This  program,  known  as  the  Human  Awareness  Course,  underwent  a series  of  changes  in 
orientation,  content,  and  methods  over  time. 

At  the  inception  of  the  contract,  the  Human  Awareness  Course  was  under  develop- 
ment as  a vehicle  for  increasing  the  knowledge  of  first-  and  second-line  Navy  civilian  supervisors 
concerning  the  dynamics  of  sexism  and  racism,  with  implications  for  behavior  change  to  reduce, 
and  eventually  eliminate,  arbitrary  discrimination  in  personnel  decisions.  At  the  outset,  the 

course  fell  into  Stage  Two  of  the  category  system  described  in  Chapter  I.  At  that  time  (sum- 

1 2 

mer  1975),  the  program  was  described  by  its  authors  as  having  three  major  components: 


^ The  name  of  the  agency  has  since  been  changed  to  Office  of  Civilian  Personnel  (OCP). 
^Meeting  held  at  OCMM  offices  on  12  August  1975. 
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1.  Identification  of  the  problem  (sexism  and  racism)— Define  what 
these  things  are,  how  they  came  to  be,  and  what  can  be  done 
about  them. 

2.  Identification  of  reasons  why  the  Navy  must  deal  with  the 
problem— Deal  with  productivity,  morale,  friction,  hidden  costs 
of  discrimination;  familiarize  participants  with  the  Self-Evaluation 
Questionnaire  as  a management  tool. 

3.  Explanation  of  Navy  and  Federal  laws  and  regulations— Identify 
pertinent  statutes  and  directives;  explain  the  development  and 
EEO  implications  of  Affirmative  Actions  Plans;  etc. 

The  total  orientation  was  an  educational  one,  with  the  objective  of  changing  behavior,  and 
emphasis  on  practical  consequences. 

At  the  outset  of  the  contract,  the  objectives  of  the  Human  Awareness  Course  had 
been  defined  only  implicitly,  by  the  content  of  the  various  modules  incorporated  in  the  course. 
The  intent  of  the  course  was  apparently  to  do,  at  a minimum,  the  following  things: 

• increase  the  level  of  awareness  of  civilian  supervisors  about 
the  historical  picture  of  the  treatment  of  minorities  and 
women  in  the  U.  S.  work  force; 

• present  a theory-based  explanation  of  the  nature  of  individ- 
ual prejudice  and  discrimination; 

• describe  some  common  myths  and  stereotypes  about  women 
and  minorities  in  the  work  force,  and  refute  them  with  em- 
pirical evidence; 

• make  supervisors  aware  of  the  nature  and  objectives  of 
Affirmative  Actions  Plans; 

• inform  supervisors  about  some  methods  of  diagnosing  the 
presence  of  racial  and  sex  imbalances,  inequities,  or  per- 
ceived problems  within  their  work  units,  focusing  on  the 
Self-Evaluation  Questionnaire  (SEQ);  and 

• inform  supervisors  of  policies,  procedures,  and  programs 
which  could  be  used  to  reduce  institutional  discrimination 
in  the  Navy  civilian  work  force. 
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In  an  effort  to  clarify  and  make  explicit  the  program’s  objectives  at  that  time, 

HSR  staff  reviewed  the  detailed  lesson  plans  being  used  then  and  formulated  88  statements 
which  seemed  to  represent  implicit  objectives.  These  statements  ranged  from  highly  specific 
(e.g.,  objectives  referring  to  understanding  of  specific  training  programs)  to  very  general  (e.g., 
objectives  referring  to  movement  toward  parity  for  women  and  minorities). 

This  list  of  88  statements  was  then  reviewed  by  each  of  the  Navy  personnel  directly 
involved  in  the  development  of  the  Human  Awareness  Course.  Each  of  these  twelve  individuals 
rated  each  of  the  objectives  on  a four-point  scale,  as  follows: 

1 = This  is  an  extremely  important  objective 

of  the  Human  Awareness  Course. 

2 = This  is  a moderately  important  objective 

of  the  Human  Awareness  Course. 

3 = This  is  an  objective  of  minor  importance. 

4 = This  is  not  an  objective  of  the  Human 

Awareness  Course. 

There  was  no  single  objective  on  which  there  was  unanimous  agreement  as  to  its  importance. 
There  was,  however,  a kind  of  ordering  of  objectives  from  the  general  to  the  specific,  the  more 
specific  the  statement  the  less  important  it  was  perceived  to  be.  Appendix  A contains  an  ordered 
listing  of  the  statements  from  most  to  least  important. 

Because  of  the  overall  orientation  on  which  the  course  content  was  based,  the  objec- 
tives statements  were  phrased  in  terms  of  desired  cognitive  and  behavioral  outcomes.  Very  few 
carried  even  the  slightest  implication  of  attitude  change  as  an  objective,  and  these  usually  were 
phrased  in  terms  of  attempting  to  motivate  the  supervisor  to  undertake  a certain  type  of  action, 
which  might,  but  does  not  necessarily,  indicate  that  attitude  change  is  a goal. 

To  make  these  objectives  statements  maximally  applicable  to  the  program  as  it  is  now 
envisioned,  it  might  help  to  classify  them  as  “ultimate,”  “intermediate,”  and  “immediate” 
objectives.  To  begin  with,  there  are  several  objectives,  the  more  general  ones,  which  fall  into 
the  “ultimate  objective”  category,  measurable  in  terms  of  numerical  indicators  of  institutional 
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discrimination.  Included  here  are  all  those  goals  which  refer  to  parity  for  minorities  and 
women— parity  across  all  GS  and  Wage  Board  grade  levels,  across  all  career  areas,  in  new  hires, 
in  promotion  rates,  in  assignment  to  training  programs,  in  awards,  in  the  awarding  of  outstanding 
performance  ratings,  and  in  the  administration  of  disciplinary  actions. 

The  category  of  “interim  objectives,”  i.e.,  changes  in  patterns  of  behavior  employed 
in  decisionmaking  process,  would  include  such  statements  as  those  referring  to:  actions  the 
supervisor  can  take  to  relate  the  Affirmative  Actions  Plan  to  his  or  her  area  of  authority;  de- 
creasing the  incidence  of  various  negative  and  discriminatory  acts  on  the  job;  utilization  of  the 
various  EEO  programs  (Federal  Women’s  Program,  Spanish-Speaking  Program  etc.)  and  training 
programs  (Apprentice  Program,  Administrative  Co-Op  Program,  Worker-Trainee  Opportunities 
Program,  Upward  Mobility  Program,  etc.);  and  ability  to  develop  an  Affirmative  Actions  Plan, 
if  necessary,  including  establishment  of  numerical  goals. 

Those  objectives  which  can  be  classed  as  “implicit  intermediate  objectives,”  i.e., 
motivating  the  supervisor  or  manager  to  utilize  newly  acquired  knowledge  and  skills,  would 
include  the  most  general  kind  of  motivational  goals;  e.g.,  motivating  program  participants  “to  do 
all  they  can  to  provide  equal  employment  opportunity  for  all,”  as  well  as  more  specific  kinds  of 
motivation,  for  example:  to  attempt  to  achieve  numerical  goals  set  for  them;  to  assess  the  EEO 
performance  of  subordinates  realistically;  to  give  continuous  attention  to  measurement  of  EEO 
progress,  to  be  intimately  iamiliar  with  applicable  AAP’s;  to  be  alert  to  signs  of  prejudice  among 
subordinates;  and  to  use  existing  programs  to  best  advantage. 

Finally,  objectives  concerning  knowledge  and  perceptions,  classed  as  “explicit 
instrumental  objectives,”  would  include  all  those  objectives  stressing  awareness,  understanding, 
and  recognition  of:  such  concepts  as  racism,  sexism,  prejudice,  discrimination,  numerical  goals, 
and  affirmative  action;  the  relationship  between  the  individual  supervisor’s  behavior  and  the 
overall  EEO  picture;  the  goals  of  specific  EEO  programs  and  training  programs  and  the  appro- 
priate utilization  of  those  programs;  EEO  statutes,  regulations,  and  official  policy  governing  per- 
sonnel decisions;  myths  and  facts  concerning  working  women  and  minorities  in  the  work  force; 
and  so  forth. 

Note  that  a specific  object  or  concept  may  relate  to  an  objective  in  any  and  all  of 
these  categories.  For  example,  the  Worker-Trainee  Opportunities  Program  has  knowledge 
objectives— what  is  it,  what  are  the  eligibility  requirements;  perceptual  objectives— is  it  a 
program  with  a bad  reputation,  and  if  so,  what  can  be  done  to  change  that  perception; 
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motivational  and  behavioral  objectives— can  more  supervisors  be  motivated  to  use  the  program 
for  EEO  objectives;  and  finally,  institutional-level  objectives— does  utilization  of  the  program 
reflect  a movement  toward  parity  for  women  and  minorities;  and  what  career  progression 
impacts  does  such  increased  utilization  bring  about? 

Plans  for  implementation  of  the  program  were  not  firm  at  that  early  stage,  beyond 
the  desire  to  conduct  a field  test.  No  decision  had  been  made  as  to  how  the  program  would 
be  implemented  on-site  once  development  was  completed.  Two  options  were  mentioned,  a 
traveling  team  to  visit  various  commands  and  activities  on  a rotating  basis;  and  a procedure 
whereby  program  participants  would  travel  to  a central  location,  perhaps  Washington,  D.  C, 
for  the  required  three-day  period.  Also  indeterminate  was  the  identity  of  the  personnel  who 
would  conduct  the  program,  although  the  use  of  a specially-trained  team,  as  opposed  to  use  of 
the  program’s  authors,  appeared  more  feasible. 

A field  test  of  this  preliminary  version  of  the  program  was  conducted  at  the 
Philadelphia  Naval  Shipyard  during  the  period  3-5  September  1975.  The  course  of  instruction 
was  presented  by  the  authors  of  the  five  modules  then  in  existence.  Participants  were  about 
30  first-  and  second-line  supervisors  from  a number  of  separate  Navy  activities.  They  represented 
a variety  of  job  series— clerical,  blue-color,  technical,  and  professional-and  included  men  and 
women,  with  a few  military  personnel  who  were  supervisors  of  civilian  personnel  taking  part. 
Several  of  the  participants  had  primary  or  ancillary  duties  as  EEO  coordinators,  Federal 
Women’s  Program  Coordinators,  etc. 

The  program  was  well-received  by  participants,  with  only  a relative  few  expressing 
any  more  than  mild  dissatisfaction,  and  then  on  specific  aspects  of  the  two-and-a-half  day 
experience  rather  than  overall  negative  feelings.  Based  on  feedback  from  participants  and  on 
the  perceptions  of  the  program  staff,  it  was  decided  to  make  some  modification  of  the  program 
and  begin  implementation  on  a pilot  basis  soon  thereafter. 

On  the  basis  of  this  information,  HSR  set  out  to  achieve  the  contract  objectives 
stated  above.  Even  as  this  was  occurring,  however,  policy  deliberations  were  taking  place 
within  the  Navy  which  were  to  have  a profound  effect  on  the  further  development  of  the 
Human  Awareness  Course.  As  early  as  the  Fall  of  1975,  the  Chief  of  Naval  Personnel  was  de- 
scribed^ as  having  an  interest  in  developing  an  exchange  of  EEO  personnel  and  of  training 

^Source:  Meeting  at  OCMM,  12  August  1975. 
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experiences  across  military-civilian  lines  in  the  Navy.  The  military  segment  of  the  Navy  at 
that  time  had  considerably  more  experience  with  human  relations  training  than  did  the  civil- 
ian side,  in  the  form  of  the  UPWARD  program,  then  about  to  enter  Phase  II.  The  UPWARD 
program  for  active  duty  Navy  personnel  began  as  a Stage  One  type  of  program  aimed  at  a 
global  kind  of  interracial,  interethnic,  intercultural  “awareness.”  By  the  time  UPWARD  had 
reached  a majority  of  its  target  audience,  viz.,  fleet  personnel,  a change  to  a more  institutional- 
oriented  educational  program  of  the  Stage  Two  type  was  contemplated. 

The  Navy’s  Phase  II  Program  was  planned  along  the  lines  of  a systems  improvement 
organizational  development  model.  A team  of  specialists,  known  as  an  Equal  Opportunity 
Assistance  Team  (EOAT),  was  to  be  employed  to  travel  to  each  command  to  implement  the 
program.  After  a thorough  study  of  a given  command’s  structure  and  functions,  the  team 
would  visit  the  command,  perform  certain  diagnostic  operations  on-site,  in  conjunction  with 
command  personnel,  and  provide  feedback  to  the  command  structure.  A command  planning 
meeting  was  then  to  be  held  to  develop  a plan  of  action  for  correcting  whatever  deficiencies  had  been 
identified.  Meanwhile,  the  EOAT  would  conduct  training  sessions  for  selected  command  personnel 
so  that  local  personnel  would  have  the  requisite  knowledge  to  conduct  workshops  within  the  com- 
mand. This  last  step  was  an  effort  to  insure  command  self-sufficiency  in  the  HR/EO  area. 

Phase  II  was  scheduled  to  include  the  shore  establishment  as  well  as  the  fleet.  There 
are  some  major  differences  between  the  fleet  and  the  shore  establishment,  however,  that  car- 
ried implications  for  this  Phase  II  program.  These  included: 

• the  smaller  number  of  minority  personnel  on  shore  as  com- 
pared to  the  fleet; 

• the  presence  of  a majority  (approaching  75%)  of  civilians  in 
the  shore  establishment  whereas  the  fleet  is  almost  exclusively 
military;  and 

• the  generally  higher  grade  level  of  shore  personnel  compared  to 
sea  duty  personnel. 

These  last  two  factors  provide  a bridge  between  Phase  II  and  the  Human  Awareness  Course, 
and  would  appear  to  be  the  basis  for  a more  unified  military-civilian  EO/EEO  program, 
and  for  the  move  toward  a program  more  oriented  toward  supervisors  and  managers. 
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Over  the  intervening  time  period,  a number  of  policy  decisions  have  been  made 
which  will  ultimately  result  in  such  a unified  program.  Although,  at  this  writing,  the  final 
form  of  that  program  is  not  yet  fixed  in  all  details,  Figure  1 2 illustrates  the  nature  of  the 
changes  undergone  by  the  original  Human  Awareness  Course  from  its  beginning  to  the  present 
time  (spring  1977). 
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Figure  12. 

Modification  in  Process  and  Content  Orientation 
of  the  “Human  Awareness  Course” 


Process 


Human  Awareness  Course 
(Fall  1975) 

1 . Self-contained  program 

2.  Educational  orientation 

3.  Pre-packaged  program 

4.  Program  “imposed”  on 
the  command 

5.  Limited  command  in- 
volvement 

6.  Program  staff  undefined 


Shore  Equal  Opportunity  Program 
(Spring  1977) 

-►  One  segment  of  an  overall  organi- 

zational development  process 

-►  Training  orientation 

-►  Program  tailored  to  dynamics  within 

the  command 

V Command  participation  and  self- 

sufficiency 

Command  participation  in  diagnosis, 
planning,  decisionmaking,  implementation 

"►  Program  conducted  by  specially  trained 

traveling  teams 


Content 


1.  Civilian  orientation  only  ► 

2.  Global  “awareness”  focus  ^ 

3.  Primary  orientation  toward  ► 

individual 

4.  Theoretical  ► 

5.  Historical  ► 

6.  Academic  approach  to  insti-  ► 

tutional  measures  of  dis- 
crimination 


Combined  military-civilian,  with  selec- 
tion of  appropriate  modules  for  each 
command  separately 

Emphasis  on  action-orientation, 
supervisor/manager  behavior 

Primary  orientation  toward  the  individ- 
ual’s role  within  the  institutional  frame- 
work 

Practical/applied 

Contemporary 

Practical  approach  to  discrimination 
and  the  implications  of  personnel 
decisions. 


7.  Implicit  distinction  based  on  ► 

“morality”  (right  versus  wrong) 


Distinction  based  on  “legality”  (legit- 
imate versus  legally  challengable  behavior) 


8.  Focus  on  sexism  and  racism 


Focus  on  all  forms  of  arbitrary  discrim- 
ination; e.g.,  that  based  on  age,  religion, 
etc.,  as  well  as  sex  and  race. 
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CHAPTER  V 


APPROACH  TO  IMPACT  ASSESSMENT 


In  a discussion  of  the  growing  interest  in  and  emphasis  on  evaluation  as  an  area 
of  specialization  within  the  discipline  of  psychology,  Sommer  ^ 4 differentiates  the  objec- 
tives of  “research”  from  those  of  “evaluation”  and  concludes  that  the  two  are  distinctly 
different.  A study  can,  however,  have  both  reserach  and  evaluation  aspects  within  the 
same  design,  using  a single  method.  The  present  study  is  an  example  of  that  combinatioa 
In  it,  the  impact  assessment  ultimately  hinges  on  the  long-range  changes  which  occur  in 
patterns  of  employment.  Interim  stages  of  assessment,  however,  can  be  used  to  provide 
feedback  to  the  program  concerning  what  works  and  what  doesn’t,  under  what  conditions, 
for  what  types  of  people,  in  terms  of  impacts  on  individual  knowledge,  perceptions,  moti- 
vation, behavior,  and  even,  perhaps,  attitudes.  Attitudes,  in  fact,  seem  to  be  a particularly 
notable  example  of  a research  focus,  since  the  policymakers  for  the  program  have  explicitly 
denied  that  attitude  change  is  an  objective.  If  one  employs  a particular  design  and  a specific 
methodology  to  assess  long-range  institutional  impacts,  why  cannot  the  identical  design 
and  method  be  used  to  research  more  immediate  questions,  to  test  hypotheses,  so  as  to 
enhance  the  chances  of  achieving  long-range  impacts? 

This  is,  in  fact,  the  approach  taken  here.  Measures  have  been  suggested  for 
assessing  long-range  institutional  impacts,  and  for  assessing  immediate  and  intermediate 
impacts.  These  latter  measures  are  also,  in  effect,  tests  of  hypotheses  concerning  condi- 
tions of  training.  In  addition,  measures  of  attitude  change  are  suggested  purely  as  means 
for  potential  testing  of  hypotheses  concerning  immediate  and  long-range  effects  of  cognitive, 
perceptual,  and  behavior  oriented  training  on  individual  attitudes.  In  summary,  research  and 
evaluation  measures  are  combined. 

With  regard  to  measures  of  institutional  discrimination,  little  is  said  here  about 
the  content  of  such  measures  because  the  Equal  Opportunity  Project  Office  has  developed 


l4Robert  Sommer,  “No,  Not  Research,  I Said  Evaluation!”  APA  Monitor , 8,  4 (April  1977),  p.  1. 
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a set  of  numerical  indicators  of  EEO  status.  Ways  of  displaying  and  interpreting  these  indi- 
cators are  discussed.  However,  primary  emphasis  is  on  measures  of  impact  on  the  individual. 

Assessment  of  Individual  Impacts 

The  evaluation/research  measures  of  impacts  on  individual  participants  in  which 
there  is  interest  here  are  behavior,  knowledge,  perceptions,  and  attitudes.  In  each  case,  the 
objective  is  to  determine  if  and  in  what  ways  the  training  experience  affected  measures  of 
these  variables.  Information  from  the  analysis  of  impacts  will  be  fed  back  to  policymakers 
and  to  those  personnel  who  dictate  the  way  in  which  the  program  is  implemented.  To  pro- 
vide appropriate  policy  and  procedural  feedback,  the  analysis  must  be  done  in  a systematic 
way. 

Selection  of  a Design 

The  alternative  evaluation/research  designs  available  to  evaluators  of  the  SEOP 
are  somewhat  limited.  Using  the  Campbell  and  Stanley  ^ classification  of  designs  for  research, 
the  acceptable  alternatives  are  limited  to  the  following: 

• Design  7.  The  Time-Series  Experiment 

• Design  10.  The  Non-Equivalent  Control  Group  Design 

• Design  14.  The  Multiple  Time-Series  Design. 

All  three  of  these  designs  are  “quasi-experimental”  in  nature  in  that  either  no  control  group 
is  employed  (Design  7)  or  random  assignment  of  individuals  to  experimental  and  control 
conditions  is  not  feasible  (Designs  10  and  14). 

The  equivalent  control  group  problem  is  a common  one  in  social  research.  In  the 
case  of  the  SEOP,  it  is  not  feasible  to  enter  a Navy  activity  and  randomly  select  half  of  the 
supervisors  and  managers  to  receive  EEO  training  and  half  to  receive  no  training,  for  several 
reasons: 


^Campbell  and  Stanley,  1963 ,op.  cit. 
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1 . Because  there  is  likely  to  be  quite  a bit  of  interaction  among 
supervisors  and  managers  within  an  activity  and  especially 
within  a unit,  the  effects  of  no  training  would  probably  be 
contaminated  as  the  trained  and  untrained  personnel  com- 
municate about  the  training  program. 

2.  Institutional  measures  of  impact  would  be  difficult,  if  not 
entirely  impossible,  to  aggregate  in  terms  of  the  two  randomly 
selected  groups. 

3.  Similarly,  because  of  the  drastically  reduced  frequency  of  occur- 
rence of  the  decisions  which  would  be  reflected  in  institutional 
statistics,  either  much  smaller  (and  less  stable)  frequencies 
would  be  analyzed,  or  the  period  of  time  over  which  data  were 
aggregated  would  necessarily  be  extended. 


These  factors  are  in  addition  to  other  considerations,  either  logistic  or  ethical  (e.g.,  the  “right- 
ness” of  withholding  EEO  training  from  one  group  when  the  resources  are  readily  available 
and  in  operation). 

Since  Design  14  contains  the  essential  features  of  the  other  two  designs,  it  should 
be  given  first  consideration.  This  design  can  be  summarized  as  follows: 


What  is  required  here?  First,  a control  group  must  be  identified  which  is  similar  to  the  “experi- 
mental” group  in  all  essential  aspects.  This  means  that,  when  training  is  to  occur  within  a 
shipyard  which  does  maintenance  on  sea-going  vessels,  and  has  a preponderance  of  personnel 
in  job  series  related  to  that  function,  the  appropriate  comparison  is  with  another  shipyard 
with  a similar  job  series  structure  rather  than  with  an  air  station,  a personnel  research  center, 
an  engineering  activity,  etc.  At  a minimum,  it  is  recommended  that  groups  be  matched  on: 
type  of  activity;  nature  of  the  function  performed;  job  series  represented;  and  minority  and 
female  population  sizes.  The  doubtful  feasibility  of  this  type  of  matching  process  for  more 
than  a few  Navy  activities  is  the  biggest  drawback  to  the  use  of  Design  14. 


Group  1 Pre-Test 

Group  2 Pre-Test 


Training  Post-Test  Followup 

— Post-Test  Followup 
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Assuming  that  matching  is  possible,  however,  the  next  step  is  to  compare  the 
selected  activities  in  terms  of  the  pre-test  scores  on  the  variables  selected  as  impact  assess- 
ment measures.  If  they  differ  significantly  on  these  measures,  further  steps  must  be  taken; 
e.g.,  assessment  of  pre-test/post-test  change  by  covarying  out  the  pre-test  scores  from  the 
post-test  scores.  In  essence,  however,  if  the  pre-test  difference  is  large,  the  comparison 
should  not  be  made  because  the  groups  do  not  approach  any  semblance  of  equivalence, 
which  is,  after  all,  the  purpose  of  matching. 

If  the  activities  can  be  matched,  and  if  the  pre-test  scores  are  not  significantly 
different,  the  comparison  should  be  made.  Note  that  in  the  design  as  depicted  above,  two 
post-training  measures  have  been  taken.  One  might  be  taken  immediately  after  completion 
of  training,  to  detect  any  immediate  changes.  This  would  involve  only  measures  of  knowl- 
edge, perceptions,  and  attitudes,  however,  since  behavioral  and  institutional  change  would 
not  yet  have  had  an  opportunity  to  occur.  A second  post-training  followup  measure  on  all 
variables  might  then  be  taken  six  months  later,  to  detect  the  durability  of  any  immediate 
change  and  the  onset  of  behavioral  change.  For  research  purposes;  e.g.,  to  describe  the  course 
of  any  individual  change  over  time,  additional  post-training  measures  might  be  added,  either 
within  the  first  six  months  or  thereafter,  or  both.  Care  must  be  taken  with  regard  to  such 
factors  as  respondents’  “learning  the  test,”  or  poor  test-retest  reliability  of  the  instruments, 
however,  if  this  is  to  be  done.  Problems  of  this  nature  can  be  solved  by  extensive  instru- 
ment development  work  to  insure  high  reliability,  by  using  two  or  more  equivalent  forms 
of  the  instruments,  or  by  randomly  assigning  the  participants  to  subgroups  for  retesting 
at  different  time  intervals. 

The  shapes  of  the  time-series  lines  over  time  would  be  expected  to  differ  some- 
what from  one  measure  to  another.  Knowledge  of  EEO-related  facts  might  be  expected 
to  take  an  immediate  jump  during  the  training  period,  level  off  over  the  next  few  months 
(or  increase  slightly  as  awareness  of  EEO  is  heightened),  and  perhaps  drop  off  gradually  over 
time  as  certain  little-used  facts  are  forgotten.  Perceptual  measures  would  probably  show  a 
slight  increase  at  the  first  post-test,  a further  increase  over  the  next  six  months,  and  a 
leveling  off  thereafter.  Attitude  change  might  occur  abruptly  for  some  individuals,  gradually 
for  others,  and  not  at  all  for  still  others.  Whatever  the  temporal  course  of  the  change,  how- 
ever, the  evaluator  is  still  interested  in  the  basic  questions: 
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• Is  change  occurring  as  a result  of  training? 

• Does  the  change  show  a systematic  variation;  e.g.,  is  one  area 
of  factual  knowledge  systematically  lower  in  gain  than  others? 

• What  are  the  implications  of  the  nature  of  this  change  for 
policy  and  for  procedure? 

Design  1 0 is  similar  to  Design  1 4 in  that  the  two  groups  cannot  be  considered 
equivalent.  Design  1 0 is  based  on  the  assumption  of  a single  pre-test  and  a single  post-test; 
to  extend  it  to  multiple  post-tests  is,  in  fact,  to  convert  it  to  Design  14.  Thus,  Design  14  is 
superior  if  it  is  important  to  consider  changes  over  time,  as  it  certainly  would  seem  to  be 
in  the  case  of  institutional  statistics  and  individual  measures  as  well,  assuming  that  feedback 
is  desired  concerning  the  optimum  time  period  for  a “refresher  course,”  etc. 

Design  1 0 can  be  depicted  as  follows: 

Group  1 Pre-Test  Training  Post-Test 

Group  2 Pre-Test  — Post-Test 

Design  7 has  the  advantage  of  including  several  points  in  time  in  the  analysis,  but 
lacks  even  the  precaution  of  a non-equivalent  control  group  contained  in  Design  1 0.  Design  7 
takes  the  following  form: 

Group  1 Pre-Test  Training  Post-Test  Followup 

Once  again,  as  many  followup  measures  as  desirable  can  be  taken. 

To  summarize  then,  it  appears  not  to  be  feasible  to  employ  a true  experimental 
evaluation  design  for  the  SEOP.  Of  the  quasi-experimental  designs  available,  the  Multiple 
Time-Series  Design  is  preferable  because  it  allows  attribution  of  any  measured  change  to 
the  training  program  by  virtue  of  the  inclusion  of  a control  group;  it  also  allows  for  observa- 
tion of  the  course  of  change  in  the  impact  measures  over  time.  The  Time-Series  Experiment 
allows  for  over-time  observations  but  lacks  a control  group.  And  the  Non-Equivalent  Control 
Group  Design  is  a foreshortened  version  of  the  Multiple  Time-Series,  with  the  virtue  of  a 
control  group,  but  no  temporal  factor  built  in. 
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Selection  of  Appropriate  Impact  Measures 


In  Part  I of  this  report,  it  was  recommended  that  a detailed  statement  of  objectives 
be  prepared  early  in  the  course  of  development  of  a program,  and  that  this  set  of  objectives 
be  used  as  a basis  for  criterion  measures  of  program  impact.  At  the  time  of  this  writing,  no 
such  detailed  listing  of  objectives  is  available  for  the  SEOP;  however,  a set  of  objectives  has 
been  inferred  from  several  project  documents.  These  appear  in  Figure  13.  On  the  basis  of 
these  objectives,  and  on  the  basis  of  extensive  study  of  the  objectives  and  actual  content  of  a 
much  earlier  version  of  the  Human  Awareness  Course,  a number  of  items  have  been  written 
to  assess  the  effects  of  the  EEO  module  of  the  SEOP  on  individual  supervisors  and  managers. 
These  items  appear  in  a companion  document^  to  this  report.  They  are  separated  into 
measures  of  behavior  (self-reports),  knowledge,  perceptions,  and  attitudes.  (The  latter  two 
variables  are  measured  by  several  “scales”  which  require  further  psychometric  development, 
but  most  items  of  which  have  been  used  in  otiier  similar  contexts.) 

Assessment  of  Institutional  Impacts 

The  SEOP  Project  staff  within  OCP  have  identified  a set  of  some  25  statistical 
reports  of  EEO  status  within  a Navy  activity  which,  in  conjunction  with  several  other  non- 
statistical  sources  of  information,  provide  the  basis  for  diagnosing  EEO  imbalances  and 
inequities  within  each  separate  activity.  Rather  than  review  the  list  of  statistical  indicators 
here,  it  will  be  assumed  that  these  were  selected  to  cover  all  the  decision  areas  which  relate 
to  the  institutional-level  objectives  of  the  SEOP,  and  that  they  represent  the  best  available 
information  on  the  following  areas: 

• Selection  of  personnel  to  fill  vacant  positions. 

• Decisions  concerning  advancement  of  an  individual 
within  the  organization. 

• Decisions  relating  to  job  reclassification,  either 
upgrading  or  downgrading. 

• Selection  of  individuals  for  specific  training  oppor- 
tunities. 

^6Dale  K.  Brown,  Critique  of  the  Shore  Equal  Opportunity  Program  (SEOP),  (McLean,  Va.: 
Human  Sciences  Research,  Inc.,  May  1977),  Technical  Note.  (In  Review.) 
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Figure  13. 

Examples  of  SEOP  Objectives 


Knowledge  Objectives— The  SEOP  is  intended  to  convey  certain  factual  information 
to  the  individual  supervisor  or  manager.  As  a result  of  participation,  the  supervisor  or 
manager  should  be  able  to  understand  the  following  things  (and  recognize  examples,  where 
applicable): 

• Institutional  discrimination. 

• Individual  discrimination. 

• Title  VII  of  the  Civil  Rights  Act  of  1964  (as  amended,  1972). 

• Possible  discriminatory  effects  of: 

a.  the  use  of  tests  in  employee  selection; 

b.  the  use  of  qualification  requirements  which  have  discrimi- 
natory side  effects; 

c.  educational  requirements; 

d.  documentation  of  promotion  actions; 

e.  the  composition  of  selection  panels; 

f.  use  of  non-job-related  qualifications; 

g.  organizational  policies  such  as  short  lead-time  for  Personnel 
Office  to  fill  a position,  no-transfer  policy  between  units, 
promotion  from  within,  etc.; 

h.  protective  attitudes  toward  women  and  minorities; 

• The  legal  ramifications  of  all  of  the  above  practices  and  policies/ 

• Affirmative  Actions  Plans. 

• The  difference  between  a goal  and  a quota. 

• Requirements  for  supervisor/manager  performance  in  EEO. 

• Objectives,  structure  and  function  of  the  EEO  Program. 

• Appropriate  and  inappropriate  uses  of  EEO  office. 

• Types  of  training  programs  available  which  can  be  used  to  counteract 
institutional  race  and  sex  imbalances,  e.g.,  Upward  Mobility,  Worker- 
Trainee  Opportunities,  Apprentice,  and  Admin  Co-Op  Programs, 
and  the  requirements  for  admission  to  these. 

• Current  relative  maldistribution  of  women  and  minorities  across  grades 
and  job  series. 

• Demographic  statistics. 

• Concentration  statistics. 

• Comparative  statistics. 

• Etc. 
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Figure  13  (Continued) 


Perceptual  Objectives-The  SEOP  is  intended  to  increase  the  accuracy  of  the 
supervisor/manager’s  perceptions  concerning: 

• Objectives  of  the  EEO  program. 

• EEO  implications  of  all  personnel  decisions. 

• The  presence  of  institutional  discrimination  in  the  Navy  civilian 
work  force. 

• The  capabilities  of  women  in  comparison  to  men,  and  of  minorities 
in  comparison  to  whites. 

• Navy  commitment  to  EEO. 

• The  scope  of  the  supervisor/manager’s  responsibilities  for  EEO. 

• The  difference  between  “affirmative  action”  and  “reverse 
discrimination.” 

• Etc. 

Behavioral  Objectives— The  SEOP  is  intended  to  increase  the  occurrence  of  be- 
haviors of  the  following  types  among  supervisors  and  managers. 

• Review  of  current  personnel  policies  and  procedures  to  identify 
EEO  shortcomings  and  implications. 

• Development  of  a personal  statement  of  objectives  regarding 
EEO. 

• Periodic  scheduled  review  and  planning  for  future  personnel 
decisions  of  all  types. 

• Maintenance  of  detailed  documentation  regarding  each  personnel 
decision  in  which  the  supervisor  or  manager  participates. 

• Allowing  a reasonable  amount  of  lead  time  in  which  to  identify 
qualified  candidates  for  vacant  positions. 

• Periodic  scheduled  review  of  institutional  statistics  to  identify 
areas  of  sex  or  race  imbalance. 

• Formal  requests  to  Personnel  Officer  that  women  and/or  minority 
candidates  be  identified  for  vacant  positions. 

• Etc. 
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• Selection  of  training  programs  for  specific  individuals. 

• Utilization  of  the  skills  acquired  through  training. 

• Duty  assignments. 

• Disciplinary  actions. 

• Routine  evaluation  of  the  performance  of  subordinates. 

• Recommendations  for  awards,  rewards,  commendations, 
and  other  positive  acts. 

• Reduction-in-force  (RIF)  decisions. 

• Termination  of  employment. 

What  is  recommended  for  consideration,  however,  is  a method  for  computing  and 
displaying  institutional  statistics  to  make  them  maximally  useful  to  the  supervisor,  or  EEO 
specialist.  The  easier  a particular  statistic  is  to  comprehend,  the  more  efficient  is  the  process  of 
diagnosing  and  assessing  the  presence  of  a problem.  The  method  used  to  display  a statistical 
measure  can  enhance  the  ease  of  interpretation  considerably.  For  a supervisor  or  manager  to  be 
forced  to  give  thorough  consideration  to  numerous  multi-page  computer  printouts,  having 
obscure  or  cryptic  labels,  and  numerous  rows  and  columns  of  frequencies  and  percentages,  is  to 
place  an  unnecessary  burden  on  that  individual,  and  runs  the  risk  of  increasing  the  tendency 
already  present  in  some  people  to  resist  “the  EEO  numbers  game.” 

Human  Sciences  Research  has  developed  an  index  of  representativeness  (the  “Dif- 
ference Indicator,”  D.I.)  for  use  by  the  U.S.  Army  which  converts  institutional  personnel 
and  decision  outcome  statistics  to  a simple  numeric  index,  easily  displayed  and  readily  in- 
terpretable as  a percentage  of  over-  or  under-representation  of  a particular  group.  * ^ The  D.I. 
can  be  displayed  graphically  in  a number  of  useful  ways.  The  report  ^ on  the  latest  de- 
velopment of  the  D.I.,  an  extension  of  the  concept  to  use  at  brigade,  battalion,  and  company 
levels,  describes  the  index,  its  computation,  its  display,  its  interpretation,  and  its  action 

17  T 

The  concept  is  similar  to  the  Equal  Opportunity  Quality  Indicator  (EOQI)  used  by  the  Navy. 

The  graphic  display  and  interpretability  of  the  D.I.,  however,  appears  more  advantageous  to  the  Navy  from 
an  efficiency  standpoint. 

18 

Nordlie  and  Edmonds,  1977,  op.  cit. 


53 


implications.  The  reader  is  directed  to  this  report  for  full  details  of  the  system.  However, 
a brief  description  and  some  illustrative  examples  of  the  computation  and  graphic  display 
of  the  D.I.  follow.  This  discussion  is  adapted  from  the  report  cited  abbve. 

The  Difference  Indicator  takes  a single  decision  area  for  a given  work  unit  over 
a specified  period  of  time  and  displays  numerically  and  graphically  the  extent  to  which  a 
specific  group  of  persons-non-whites,  for  example-  are  over - or  under-represented  in 
decision  outcomes  compared  to  what  would  be  expected  if  all  decisions  were  made  without 
regard  to  skin  color  (in  the  example)  or  another  characteristic  irrelevant  to  job  performance. 
The  Difference  Indicator  formula  is: 


The  purpose  of  multiplying  by  100  is  simply  to  put  the  results  in  percentage  form  which  is 
generally  easily  understood.  The  purpose  of  subtracting  100  is  so  that  the  indicator  will 
be  zero  whenever  the  actual  and  expected  number  are  the  same;  negative  when  the  actual 
number  is  smaller  than  expected,  and  positive  when  the  actual  number  is  larger  than  expected. 

In  order  to  use  the  Difference  Indicator  System,  certain  data  must  be  collected 
carefully  and  accurately.  To  calculate  any  D.I.,  two  numbers  are  required: 

• the  actual  number  of  minorities  or  women  on  that  dimension; 
and 

• the  expected  number  of  minorities  or  women  on  that  dimen- 


Each of  these  numbers  must  apply  to  the  same  time  period  or  point  in  time.  The  actual 
number  is  obtained  by  merely  counting  the  number  of  minorities  or  women  in  a particular 
category.  The  expected  number  is  derived  from  two  other  numbers: 


sion. 


1 . the  total  number  of  persons  “eligible”  to  be  in  that  category; 


2.  the  number  of  minorities  or  women  “eligible”  to  be  in  that 
category. 
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Eligible  means  being  in  position  or  having  a chance  to  receive  the  action  denoted  by  the 
dimension.  For  Promotion  to  Grade  WL-07,  the  eligible  population  would  be  those  in 
lower  grades  who  meet  the  minimum  requirements. 

By  dividing  (2)  above  by  (1)  above,  the  expected  percentage  of  minorities  or 
women  in  the  category  is  determined.  Multiplying  this  percentage  by  the  total  number  of 
persons  in  the  category  results  in  the  expected  number  of  minorities  or  women  in  that 
category.  An  example  of  this  is  as  follows: 

1.  Total  number  of  persons  eligible  for  promotion  to  grade  WL-07  = 

104. 

2.  Number  of  minorities  eligible  for  promotion  to  grade  WL-07  = 22. 

22 

Dividing  (2)  by  (1)  = — — = .21  or  21%  = expected  percentage. 

1 LKt 

3.  Total  number  of  persons  selected  for  promotion  to  grade  WL-07  = 

84. 

Multiply  expected  percentage  times  total  number  selected  = 

21%  x 84=  17.64. 

Expected  number  =18(1 7.64  rounded). 

Remember  that  the  expected  number  of  minorities  is  that  number  of  minorities  which  would 
occur  in  a given  category  if  skin  color  were  unrelated  to  being  in  that  category.  It  is  that 
number  which  would  occur  if  the  persons  selected  for  a particular  category  were  selected 
purely  randomly  from  the  eligible  population.  The  expected  number  has  no  meaning  other 
than  as  a reference  point  from  which  to  measure.  It  should  not  be  interpreted  as  indicating 
a goal  or  quota.  It  is  merely  a point  from  which  to  measure  and  has  the  same  meaning  wherever 
it  is  applied. 

It  can  be  seen  from  the  above  that  for  any  dimension  or  category,  four  numbers 
are  required  to  calculate  a D.I.: 
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1 . the  total  number  of  persons  actually  in  the  category; 

2.  the  total  number  of  persons  “eligible”  to  be  in  the  category; 

3.  the  number  of  minorities  “eligible”  to  be  in  the  category; 

4.  the  number  of  minorities  actually  in  the  category. 

If  the  actual  number  of  minorities  promoted”  in  the  above  example  were  10, 
the  D.I.  would  be: 

x 10cj-100«-45. 

This  means  that  minority  group  members  are  under-represented  by  45  percent  on  that 
dimension. 

The  D.I.  is  a very  flexible  index,  and  can  be  used  for  any  personnel  decision  type  for 
which  the  four  required  pieces  of  information  are  available. 

Displaying  of  the  D.I.  can  be  done  in  any  of  at  least  three  different  ways.  These  are: 

Type  I Display  — A bargraph  comparing  all  D.I.’s  for  a 

given  unit  for  a single  time  period. 

Type  II  Display  — A bargraph  comparing  all  units  in  an 

activity  on  the  D.I.  for  each  dimension 
separately. 

Type  III  Display  — A graph  comparing  D.I.’s  for  the  same 

dimension  on  the  same  unit  at  different  times. 

Examples  of  these  three  display  types  appear  in  Figures  14,  15,  and  16. 

The  different  types  are  used  for  different  purposes.  Once  the  D.I.’s  for  a given 
time  period  are  calculated  for  a particular  unit,  it  will  almost  always  be  convenient  to 
graph  these  in  a Type  I display.  This  type  of  display  is  useful  for  a supervisor/manager  to 
see  on  what  dimensions  large  minority-white  or  male-female  discrepancies  are  occurring  in 
his  or  her  unit. 

A Type  II  display  is  useful  for  a manager  to  compare  similar  subordinate  units 
on  any  particular  dimension  in  which  he  or  she  has  an  interest.  This  type  of  display  is 
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useful  for  a commander  to  determine  whether  subordinate  units  are  pretty  much  alike 
on  a particular  dimension  or  whether  there  are  large  differences  from  one  unit  to  the  next. 

A Type  III  display  is  useful  for  a supervisor  or  manager  to  evaluate  trends  over 
time.  This  display  shows  the  direction  of  changes  occurring  on  any  dimension.  It  is  a 
simple  matter  once  any  D.I.  is  calculated  to  convert  it  to  the  graphic  display.  Obviously, 
when  it  is  available,  data  for  longer  periods  of  time  than  one  year  can  be  plotted  on  similar 
charts. 

The  greatest  value  of  this  management  tool  is  realized  with  the  use  of  the  Type  III 
display  which  allows  one.  to  see  how  changes  are  occurring  on  a dimension.  Normally,  one 
is  interested  in  reducing  large  white/non-white  or  male/female  differences.  The  extent  to 
which  such  differences  are  being  reduced  can  be  determined  by  observing  the  trend  lines  on 
the  over-time  graphic  display. 

Another  factor  that  can  only  be  evaluated  from  the  over-time  display  is  the  consistency 
of  differences.  If  one  sees  a D.I.  for  some  dimension  fluctuate  around  zero-i.e.,  + 84  percent 
the  first  quarter,  -10  percent  the  second  quarter,  +5  percent  the  third  quarter,  and  -12  percent 
the  fourth  quarter— one  would  conclude  that  the  D.I.  is  essentially  zero  and  is  just  showing 
random  fluctuation  from  one  time  to  the  next.  However,  if  one  observed  a D.I.  that  varied 
between  -7  percent  and  -1 1 percent,  for  example,  even  though  those  are  relatively  small 

discrepancies,  the  fact  that  they  are  consistent  over  time  means  that  the  D.I.  for  that  dimen- 
sion is  definitely  not  zero  and  some  factor(s)  is  responsible  for  perpetuating  that  consistent 
difference. 

Establishment  of  a D.I.  system  is  not  necessarily  an  easy  matter.  Appropriate 
dimensions  must  be  identified  and  the  accretion  and  aggregation  of  the  required  data  must 
be  routinized.  However,  a data  processing  system  for  the  D.I.,  once  established,  should  run 
smoothly  and  should  remove  a large  burden  from  those  charged  with  acqusition  and  inter- 
pretation of  statistical  data  for  EEO  purposes. 

No  extensive  discussion  of  the  interpretation  of  the  D.I.  will  be  given  here,  but 
the  report  cited  above  does  deal  with  that  topic  at  length. 
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Figure  14. 

Illustrative  Example  of  a Type  I Display 
for  a Hypothetical  Work  Unit 


Time  Period:  4th  Quarter  FY ‘76  Unit:  Payroll 


Dimension  Percent  Under  Expected  No.  Percent  Over  Expected  No. 
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Figure  15. 

Illustrative  Example  of  a Type  II  Display 

Dimension:  Disciplinary  Actions -Minorities  Time  Period:  4 th  Quarter  FY ‘76 


Unit  (Hypothetical) 

Percent  Under  Expected  No. 

Percent  Over  Expected  No. 

-100  -90  -80  -70  -60  -50  -40  -30  -20  -10  C 

1 r ' ' 1 1 ' ' 1 r ! . ! 1 

) 10  20  30  40  50  60  70  80  90  100 

1.  Security 
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2.  Computer  Support 

-2i 

3.  Maintenance 

+20 

4.  Engineering 

C 

5.  Payroll 

•5  ■ 

6.  Mechanical  Shop 

■■  +10 
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Figure  16. 

Illustrative  Example  of  a Type  III  Display 


D.I.  Trend  Lines 


Dimension:  Disciplinary  Actions— Minorities  Unit:  Computer  Support 


Percent 
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Percent 
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CHAPTER  VI 


SUMMARY 


The  Shore  Equal  Opportunity  Program  (SEOP)  is  viewed  as  a forward-looking 
effort  to  eradicate  institutional  discrimination  beginning  with  its  source,  the  personnel  de- 
cisions made  on  a daily  basis  by  Navy  civilian  supervisors  and  managers. 

The  overall  assessment  of  the  SEOP  in  general,  and  the  EEO  module  in  particular, 
is  decidedly  positive,  even  at  the  intermediate  stage  of  development  of  the  program  at  the 
time  of  this  writing.  The  selection  of  a Stage  Three  Model  of  EEO  training  as  described 
above;  the  employment  of  a “total  systems”  approach,  using  an  organizational  development 
model;  the  localization  of  the  change  effort;  the  technique  of  local  command  involvement 
in  planning,  based  on  diagnosis  of  prevailing  local  conditions;  and  the  development  of  com- 
mand self-sufficiency  in  carrying  out  the  program  over  time,  all  are  seen  as  highly  praise- 
worthy aspects  of  the  SEOP.  This  approach  is  viewed  as  the  direction  of  the  future  in  EEO 
training  and  is  recommended  as  a model  for  other  organizations  and  institutions. 

A companion  document  to  this  Technical  Report,  a critique  of  the  SEOP  at  an 

1 Q 

interim  stage  of  development  identified  the  obvious  absence  of  a systematic  impact 
assessment  plan  as  the  chief  shortcoming  of  the  program.  The  present  document  is  an 
effort  to  provide  guidance  to  the  SEOP  staff  in  developing  such  an  evaluation  plan. 

The  evaluation  plan  recommended  has  the  following  characteristics: 

• It  is  suitable  for  acquiring  information  potentially  useful  to  policy- 
makers, to  human  relations  program  evaluators,  to  personnel  in- 
volved in  the  day-to-day  decisions  concerning  the  SEOP,  and  to 
those  interested  in  the  research  aspects  of  EEO-relevant  training. 

• It  employs  a scientific  approach  to  program  evaluation,  suggest- 
ing a quasi-experimental  design  of  the  Multiple  Time-Series  type. 


^Dale  K.  Brown,  1977,  op.  cit. 
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• It  incorporates  measures  of  impacts  on  the  behavior,  knowledge, 
perceptions,  and  attitudes  of  individual  supervisors  and  managers 
who  experience  the  EEO  module  of  the  SEOP,  as  well  as  group 
measures  of  perceptions  and  measures  of  institutional-level  EEO 
status. 

The  report  includes  some  examples  of  items  for  measuring  the  variables  named 
above,  within  the  context  of  the  Multiple  Time-Series  Design;  and  a new  method  of  com- 
puting and  displaying  institutional  statistics,  the  Difference  Indicator  (D.I.)  is  proposed 
for  consideration. 
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